A performance optimization method and device for a storage service, and an electronic device

Abstract

The application discloses a performance optimization method and device of a storage service, and electronic equipment, relates to the technical field of data processing, and comprises the following steps: by counting the data operation condition of a key-value storage engine, triggering data arrangement to eliminate invalid data when the condition is met, allocating differentiated execution time sequences for storage services of the same physical node to avoid resource contention, coordinating and scheduling relying on the resource state and service distribution information of a cluster management node, and then controlling the orderly execution of data arrangement. The method solves the technical problems of key-value storage engine performance fluctuation, CPU overload caused by the same node operation, and lack of unified scheduling affecting system stability, and achieves the technical effects of stabilizing storage service performance, balancing hardware resource load, and improving the availability and reliability of a distributed storage system.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, and electronic device for optimizing the performance of a storage service. Background Technology

[0002] In distributed storage systems, the process of storing / retrieving specific data in response to client requests (Object Storage Device, OSD) relies on key-value storage engines such as Rocksdb to implement data storage and read / write operations, and is widely used in data storage scenarios across multiple fields. However, existing technologies have significant drawbacks: when key-value storage engines use an LSM tree structure, data deletion is only marked rather than actually removed, leading to the accumulation of a large amount of invalid data. The data cleanup process triggered by the storage engine itself is difficult to meet storage requirements, easily causing performance fluctuations, decreased read / write efficiency, or even service crashes. If multiple storage services deployed on the same physical node perform data cleanup operations simultaneously, it can cause a sudden increase in CPU load, which can then freeze threads. Furthermore, the lack of a unified coordination and scheduling mechanism based on node resource status and storage service distribution information makes it impossible to reasonably manage data cleanup operations, affecting the overall stability and availability of the system. Summary of the Invention

[0003] This application provides a method, apparatus, and electronic device for optimizing the performance of storage services, in order to at least solve the problems in the related technologies caused by the accumulation of invalid data in key-value storage engines, such as performance fluctuations, decreased read and write efficiency and service freezes, sudden increases in CPU load caused by multiple storage services on the same node performing data processing simultaneously, and the lack of a unified coordination and scheduling mechanism affecting system stability and availability.

[0004] This application provides a method for optimizing the performance of a storage service, including:

[0005] Retrieve data operation statistics from the key-value storage engine in the storage service. When the data operation statistics meet the preset trigger threshold, trigger the data processing operation process. For multiple storage services located on the same physical node that all meet the triggering conditions, generate differentiated execution sequences to perform data processing operations on each storage service according to the execution sequence; Based on the node resource status and storage service distribution information obtained from the cluster management node, the data processing operation is coordinated and scheduled. Based on the coordination and scheduling results, control the corresponding storage service to perform data processing operations.

[0006] This application also provides a performance optimization apparatus for storage services, including: The trigger module is configured to obtain data operation statistics from the key-value storage engine in the storage service. When the data operation statistics meet the preset trigger threshold, the data processing operation process is triggered. The generation module is configured to generate differentiated execution sequences for multiple storage services located on the same physical node that all meet the triggering conditions, so as to perform data processing operations on each storage service according to the execution sequence; The scheduling module is configured to coordinate and schedule data processing operations based on the node resource status and storage service distribution information obtained from the cluster management node. The control module is configured to control the corresponding storage service to perform data processing operations based on the coordination and scheduling results.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the performance optimization method of any of the above-described storage services when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described performance optimization methods for storage services.

[0009] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-described storage service performance optimization methods.

[0010] This application addresses the challenges of performance fluctuations, decreased read / write efficiency, and service freezes caused by the accumulation of invalid data in the key-value storage engine. It also addresses the issues of sudden increases in CPU load due to simultaneous data processing by multiple storage services on the same physical node, and the lack of a unified coordination and scheduling mechanism affecting system stability and availability. This allows for the acquisition of statistical information on data operations of the key-value storage engine and the triggering of data processing procedures when a preset threshold is met. The application achieves the technical effects of stabilizing storage service performance, reasonably managing hardware resource load, and improving the overall stability and availability of the distributed storage system. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a performance optimization method for a storage service provided in this application embodiment; Figure 2 A flowchart illustrating another performance optimization method for storage services provided in this application embodiment; Figure 3 A flowchart illustrating another performance optimization method for storage services provided in this application embodiment; Figure 4 This is a schematic diagram of a performance optimization device for a storage service provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] Figure 1 This is a flowchart illustrating a performance optimization method for a storage service provided in an embodiment of this application.

[0017] like Figure 1 As shown, the method includes the following steps: Step 101: Obtain data operation statistics from the key-value storage engine in the storage service. When the data operation statistics meet the preset trigger threshold, trigger the data processing operation.

[0018] In some embodiments, during the operation of the distributed storage system's storage service, it is necessary to track various data operations of the key-value storage engine in real time to form complete data operation statistics. The key-value storage engine can adopt a data storage architecture based on a log-structured merging tree. Its data operations cover data interaction-related behaviors such as deletion, reading, and writing. The data operation statistics accurately record the execution count and related timing characteristics of each operation. To address the issue of the key-value storage engine only marking invalid data without actually removing it during deletion, leading to the accumulation of invalid data occupying storage resources and affecting read / write performance, a reasonable trigger threshold needs to be pre-set. This threshold can be adjusted according to actual storage needs and system operating characteristics, such as when consecutive deletion operations reach a certain number or the total number of read / write operations accumulates to a set value. After obtaining real-time data operation statistics, they are compared with the preset trigger threshold in real time. When the statistics meet the threshold conditions, a data cleanup operation process is immediately triggered. This process merges and compresses the data in the key-value storage engine, eliminating accumulated invalid data. This approach avoids the waste of storage resources and performance degradation caused by excessive accumulation of invalid data, effectively stabilizing the operating state of the key-value storage engine and ensuring the read / write efficiency of the storage service.

[0019] Step 102: For multiple storage services located on the same physical node that all meet the triggering conditions, generate differentiated execution sequences to perform data processing operations on each storage service according to the execution sequence.

[0020] In some embodiments, multiple storage services are typically deployed on the same physical node. A common deployment method is one storage service per hard drive, and some high-performance servers can even support dozens of storage services on a single node. When all these storage services meet the triggering conditions for data compaction operations, starting the operation simultaneously can cause a sharp spike in the node's CPU load, potentially freezing related threads and severely impacting the overall node efficiency. Therefore, it is necessary to generate differentiated execution sequences for these storage services that meet the triggering conditions. Specifically, random numbers can be selected within a preset range, each corresponding to a specific time interval. The order in which each storage service starts the data compaction operation is determined based on the differences in the random numbers, ensuring that the execution times of different storage services are staggered. Executing the data compaction operation according to this differentiated execution sequence effectively distributes CPU processing pressure, avoids resource overload, ensures the stable operation of storage services and nodes, and guarantees the orderly progress of the data compaction operation.

[0021] Step 103: Based on the node resource status and storage service distribution information obtained from the cluster management node, coordinate and schedule the data processing operation.

[0022] In some embodiments, the cluster management node, as the core of resource coordination and cluster status maintenance, is responsible for accurately acquiring node resource status and storage service distribution information, and coordinating and scheduling data processing operations based on this information. The cluster management node can comprehensively understand the real-time operating load of each node by reusing storage service status reporting messages and combining them with specialized system instructions to collect key resource usage information such as CPU and memory usage. Simultaneously, it extracts storage service distribution information from the cluster's status mapping, clearly defining the deployment affiliation of different storage services on various physical nodes and identifying which storage services reside on the same node. Considering the equality of storage services within the cluster, the cluster management node adopts an adaptive scheduling strategy, combining the remaining node resource capacity and service distribution density to rationally plan the execution order and timing of data processing operations, avoiding resource contention caused by multiple storage services triggering data processing operations simultaneously. This scheduling method allows data processing operations to be precisely matched with node resource status and service distribution, effectively avoiding resource overload, significantly improving system resource utilization, and ensuring the stable operation of the distributed storage cluster.

[0023] Step 104: Based on the coordination and scheduling results, control the corresponding storage service to perform data processing operations.

[0024] In some embodiments, after the cluster management node completes the coordination and scheduling of data processing operations, it generates a clear execution plan. This plan precisely defines the timing and order in which each storage service performs data processing operations. Based on this scheduling result, the system issues execution instructions to the corresponding storage services, controlling them to initiate data processing operations at preset appropriate time periods. Data processing focuses on merging and compressing data in storage, removing invalid data accumulated due to data deletion and mere marking, and optimizing the utilization efficiency of storage resources. During execution, the system strictly adheres to the requirements of the scheduling result, ensuring that the operation processes of each storage service do not conflict with each other. This guarantees efficient data processing without affecting the normal read and write operations of the storage services. This control method allows data processing operations to be implemented under orderly management, effectively avoiding resource contention issues, further stabilizing the operation of storage services, and improving the overall reliability of the distributed storage system.

[0025] Compared with related technologies, this embodiment obtains data operation statistics of the key-value storage engine in the storage service. When the data operation statistics meet a preset trigger threshold, a data processing operation is triggered. Differentiated execution sequences are generated for multiple storage services located on the same physical node that all meet the trigger conditions, so that data processing operations are performed on each storage service according to the execution sequence. Based on the node resource status and storage service distribution information obtained by the cluster management node, the data processing operation is coordinated and scheduled. According to the coordination and scheduling results, the corresponding storage service is controlled to perform the data processing operation. This can solve the problems in related technologies, such as performance fluctuations caused by the accumulation of invalid data in the key-value storage engine, decreased read / write efficiency and service freezes, sudden increases in CPU load caused by multiple storage services on the same node performing data processing simultaneously, and the lack of a unified coordination and scheduling mechanism affecting system stability and availability. It achieves the technical effects of stabilizing storage service performance, reasonably managing hardware resource load, and improving the overall stability and availability of the distributed storage system.

[0026] Figure 2 A flowchart illustrating another performance optimization method for storage services provided in this application embodiment includes the following steps: Step 201: Monitor and accumulate the number of deletion operations of the key-value storage engine, and / or monitor and accumulate the number of read and write operations of the key-value storage engine.

[0027] In some embodiments, during the operation of the storage service, it is necessary to continuously monitor and accurately accumulate the core data operations of the key-value storage engine. This key-value storage engine often adopts an architecture based on a log-structured merging tree. Its deletion operation does not directly remove data, but rather marks the data as invalid by adding a deletion flag. Therefore, a dedicated monitoring mechanism is needed to capture each deletion behavior. Read and write operations encompass data reading and writing, directly affecting the interaction efficiency of the storage service. By embedding a dedicated statistics module in the storage service, the execution status of each deletion, read, and write operation of the key-value storage engine is tracked in real time, and the number of each type of operation is continuously accumulated. This ensures that the statistical data fully reflects the frequency characteristics of the operations, providing accurate and reliable basic data support for subsequent threshold judgments. This approach can accurately capture the execution status of key operations, providing an accurate basis for triggering subsequent data processing operations.

[0028] Step 202: When the number of deletion operations reaches the first dynamic threshold and / or the number of read / write operations reaches the second dynamic threshold, a trigger command is generated to trigger the data cleanup operation.

[0029] In some embodiments, the first dynamic threshold and the second dynamic threshold are preset based on the hardware configuration of the storage system, the characteristics of business data flow, and actual storage needs. They can be flexibly adjusted according to different application scenarios. For example, the first dynamic threshold can correspond to a specific number of consecutive deletion operations, and the second dynamic threshold can correspond to the total number of cumulative read and write operations. After obtaining the cumulative number of deletion operations and read and write operations in step 201, the system will compare the two types of statistical data with the first dynamic threshold and the second dynamic threshold in real time. When the number of deletion operations reaches the first dynamic threshold, or the number of read and write operations reaches the second dynamic threshold, and either condition is met, the system will automatically generate a trigger command. This command will directly trigger the data processing operation process, starting the merging and compression processing of data in the key-value storage engine. This design ensures that the data processing operation starts in a timely manner at the appropriate time, effectively avoiding excessive accumulation of invalid data and ensuring the read and write efficiency and operational stability of the storage service.

[0030] Step 203: Generate a random delay parameter for each storage service that meets the triggering conditions.

[0031] In some embodiments, in the actual deployment of a distributed storage system, the same physical node often deploys multiple storage services according to storage needs. A common deployment pattern is one hard drive corresponding to one independent storage service. Some high-performance servers can even support dozens of such storage services on a single node. When these storage services all meet the triggering conditions for data processing operations within a similar time period, starting the operation simultaneously without planning will cause a sudden surge in the node's CPU load. This may not only cause related processing threads to freeze but also severely crowd out the computing resources of other normal business operations, affecting the overall stability of the node's operation. To solve this problem, a random delay parameter needs to be generated separately for each storage service that meets the triggering conditions. The value of this random delay parameter is limited to a reasonable preset range. By setting the range reasonably, it is ensured that the parameter value obtained by each storage service has a natural difference. This difference provides core support for the subsequent staggered execution sequence, effectively avoiding the situation where multiple storage services compete for CPU resources at the same time, reserving sufficient resource buffer space for the system, and ensuring that the basic operation of the node is not affected.

[0032] Step 204: Convert the random delay parameters into specific delay times according to the preset time mapping rules.

[0033] In some embodiments, the preset time mapping rule is comprehensively formulated based on the operating characteristics of the distributed storage system, the carrying capacity of the hardware devices, and the business needs under different scenarios. Its core function is to establish a clear correspondence between random delay parameters and specific delay times. This rule includes a flexibly adjustable base time interval, which users can adapt to meet the scheduling needs under different load scenarios according to actual operating conditions. The specific mapping logic can be set such that the parameter value and the delay time are linearly related. For example, when the parameter value is 0, the data processing operation is started immediately. For each increase of 1 in the parameter value, the corresponding delay time increases according to the set base time interval. If the base time interval is set to 20 minutes, then the storage service that randomly gets a value of 2 will start with a delay of 20 minutes, and the storage service that randomly gets a value of 3 will start with a delay of 40 minutes, and so on. Through such a clear and adjustable mapping rule, the abstract random delay parameter can be accurately converted into a specific delay time that can be directly executed. This ensures the differentiation of delay times between different storage services and ensures that the delay time does not exceed the carrying capacity of system resources, providing a reliable guarantee for the subsequent orderly execution of data processing operations.

[0034] In this embodiment, in order to make the delay time more closely match the real-time operating status of the node, the resource scheduling imbalance caused by the mismatch between the fixed delay setting and the node load is avoided.

[0035] The time mapping rules support dynamic configuration and can adjust the delay time according to the real-time load of the nodes.

[0036] In this embodiment, the time mapping rule supports dynamic configuration and can adjust the latency based on the real-time load of nodes. Specifically, the real-time load of nodes is determined primarily by collecting core indicators such as CPU utilization and memory usage. This real-time data can be acquired and quantitatively evaluated using specialized system commands. The time mapping rule pre-sets a basic time interval and associates it with a load threshold range and a latency adjustment coefficient. When the real-time load of a node exceeds a preset high threshold, the latency corresponding to each random latency parameter is automatically increased, further widening the execution interval between different storage services and avoiding resource contention. When the real-time load of a node is below a preset low threshold, the latency is appropriately reduced, improving the overall efficiency of data processing operations. The dynamic configuration of this time mapping rule can be achieved by issuing adjustment commands from the cluster management node, flexibly adapting to the node operation requirements under different business pressures. This approach allows the latency to dynamically match the node load, ensuring that node resources are not excessively consumed while ensuring efficient data processing operations, further improving the operational stability of the storage service.

[0037] Step 205: Determine the specific time when each storage service starts the data processing operation based on the delay time.

[0038] In some embodiments, to ensure a unified reference standard for the timing of data processing operations initiated by various storage services, the moment when each storage service first meets the data processing trigger condition is set as a unified base time. After determining the base time, the specific delay time obtained by each storage service through time mapping rules is superimposed on the base time to accurately determine the specific moment when each storage service initiates data processing operations. Since the random delay parameters corresponding to different storage services differ, the specific delay times after conversion are also different. Therefore, the start times obtained after superimposing the base time naturally form an ordered and staggered execution sequence, preventing two or more storage services from initiating data processing operations at the same time. Executing operations according to this differentiated sequence can evenly distribute the originally concentrated CPU processing pressure across different time periods, effectively avoiding resource overload issues. This ensures that data processing operations can be carried out comprehensively and efficiently, while also guaranteeing the smooth flow of normal read and write operations for storage services, allowing the system to maintain a stable operating state while completing data optimization.

[0039] Step 206: The cluster management node collects the CPU utilization and memory utilization of each physical node.

[0040] In some embodiments, the cluster management node, as the core component in the distributed storage cluster that coordinates resource scheduling and status maintenance, needs to collect the CPU utilization and memory utilization of each physical node in real time and accurately, providing reliable data support for the coordinated scheduling of subsequent data processing operations. Specifically, the cluster management node reuses the status reporting messages originally used by the storage service to report changes in the number and capacity of processed messages. By adding reporting fields for CPU and memory resource status to these messages, efficient transmission of resource data is achieved. For CPU utilization, the cluster management node sends the mpstat command to each physical node to obtain real-time CPU operating data and performs quantitative statistics. For memory utilization, it sends the free command to collect key information such as the node's memory usage and remaining capacity. This collected resource data is transmitted to the cluster management node in real time and integrated to form a resource load profile of each physical node, ensuring that the cluster management node can fully grasp the real-time operating status of each node, laying the foundation for subsequent accurate scheduling. This data collection method not only ensures the real-time and accuracy of resource information but also avoids the system overhead of building additional data transmission channels, improving overall operating efficiency.

[0041] Step 207: Based on the distributed topology of the storage service, a preset scheduling algorithm is used to plan the execution order of data processing operations for the storage service on different physical nodes.

[0042] In some embodiments, after obtaining the CPU utilization, memory utilization, and distribution topology of the storage services on each physical node, the cluster management node begins to plan the execution order of data compaction operations for the storage services on different physical nodes. The distribution topology of the storage services is derived from the cluster's state mapping, clearly defining the deployment of each storage service on different physical nodes and clarifying the storage service cluster on the same node. Considering the equal status of all storage services in the cluster, the preset scheduling algorithm prioritizes a round-robin strategy, while dynamically adapting to the real-time resource load profile of each node. For example, for physical nodes with low CPU and memory utilization and sufficient resource redundancy, the timing of data compaction operations for their corresponding storage services can be appropriately advanced in the round-robin sequence; for nodes with high resource load, the execution timing is delayed accordingly to avoid further resource overdraft. This scheduling algorithm, combining distribution topology and real-time node load, can rationally plan the execution order of data compaction for storage services on different nodes, ensuring that resource consumption on each node tends to be balanced, preventing a single node from becoming overloaded due to concentrated data compaction operations, effectively guaranteeing the overall stability of the distributed storage cluster and improving resource utilization efficiency.

[0043] To avoid the data processing operations of multiple storage services on the same physical node starting too frequently, and to prevent a sharp increase in CPU load in a short period of time that could cause thread freezing or business response delays.

[0044] The preset scheduling algorithm ensures that on the same physical node, the start time interval between any two data processing operations is not less than the preset minimum time window.

[0045] In this embodiment, the preset minimum time window is determined by comprehensively considering the hardware performance of physical nodes, the number of CPU cores, memory configuration, and the average resource consumption of data processing operations. Its duration can be flexibly adjusted according to the configuration differences of different nodes and business load requirements, for example, set to 20 minutes, to ensure that the CPU has sufficient time to complete the core process of the previous data processing task and release the occupied computing resources. During scheduling, the cluster management node first filters all storage services on the same physical node that meet the triggering conditions based on the distribution topology of the storage services. Then, based on the real-time CPU utilization and memory utilization of the node, it determines the start time of the first storage service. For subsequent data processing operations that need to be started on this node, the preset scheduling algorithm calculates the time interval between its initially planned start time and the previously determined start time. If this interval is less than the preset minimum time window, the start time of the current storage service is automatically postponed until the interval reaches or exceeds the minimum time window standard. Simultaneously, the algorithm dynamically verifies whether the adjusted start time will conflict with the execution sequence of other storage services, ensuring that the start rhythm of all data processing operations on the same node always maintains a reasonable interval. This design effectively avoids the dense triggering of data processing operations on the same physical node, further smooths the CPU load curve, and ensures that the node can respond normally to the read and write requests of the storage service while performing data processing, significantly improving the stability of node operation and the continuity of business processing.

[0046] Step 208: Based on the coordination and scheduling results, control the corresponding storage service to perform data processing operations.

[0047] For a description of step 208, please refer to the description of step 104 in the above embodiment. This embodiment will not repeat the details further.

[0048] Figure 3 A flowchart illustrating another performance optimization method for storage services provided in this application embodiment includes the following steps: Step 301: Obtain data operation statistics information of the key-value storage engine in the storage service. When the data operation statistics information meets the preset trigger threshold, trigger the data processing operation process.

[0049] Step 302: For multiple storage services located on the same physical node and all meeting the triggering conditions, generate differentiated execution sequences to perform data processing operations on each storage service according to the execution sequence.

[0050] Step 303: Based on the node resource status and storage service distribution information obtained from the cluster management node, coordinate and schedule the data processing operation.

[0051] Step 304: Based on the coordination and scheduling results, control the corresponding storage service to perform data processing operations.

[0052] For a description of steps 301-304, please refer to the description of steps 101-104 in the above embodiment. This embodiment will not repeat them in detail.

[0053] Step 305: During the data processing operation, continuously monitor the resource utilization of the system's central processing unit and memory.

[0054] In some embodiments, during data processing, to monitor the dynamic changes in system resources in real time and avoid excessive resource consumption affecting the normal operation of storage services, it is necessary to continuously monitor the CPU and memory resource utilization rates. The monitoring is coordinated by the cluster management node, relying on the established resource data acquisition mechanism. This is achieved by reusing storage service status reporting messages and combining the `mpstat` and `free` commands to obtain real-time resource utilization rates. The `mpstat` command continuously collects real-time CPU operating data to accurately track changes in its utilization rate; the `free` command captures key information such as memory usage and remaining capacity in real time and calculates memory utilization. The monitoring process employs a reasonable collection frequency to ensure timely capture of resource fluctuations. The collected CPU and memory resource utilization data are transmitted to the cluster management node in real time for recording and analysis, providing accurate and timely data support for subsequent dynamic adjustment decisions and ensuring that system resources remain under control.

[0055] Step 306: If the resource utilization rate continues to exceed the preset safety threshold, an instruction is issued through the cluster management node to dynamically postpone subsequent data processing operations that are yet to be triggered or executed.

[0056] In some embodiments, the preset safety threshold is determined by combining system hardware configuration, daily peak business load, and average resource consumption level of data processing operations. This effectively defines the safe operating range of system resources and supports flexible adjustment based on changes in actual application scenarios. When the cluster management node detects that the resource utilization rate of the central processing unit and memory continuously exceeds the safety threshold, it indicates that the current system resources are under strain. Continuing to execute subsequent data processing operations as originally planned may lead to resource overload, business response delays, or even system lag. At this time, the cluster management node dynamically calculates a reasonable delay duration based on real-time resource load and then issues a delay instruction to the relevant storage services. For storage services that have not yet triggered data processing operations, their triggering process is suspended until the resource utilization rate falls below the safety threshold before resuming. For storage services that are already scheduled for execution, their start time is postponed to ensure that the execution of subsequent data processing operations does not exacerbate resource strain. This dynamic adjustment method can flexibly adapt to real-time changes in system resources, effectively avoid the risk of resource overload, ensure the orderly progress of normal storage service operations and data processing operations, and further improve the stability and reliability of the system.

[0057] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0058] Figure 4 This is a schematic diagram of the structure of a performance optimization device for a storage service provided in an embodiment of this application, as shown below. Figure 4 As shown, it includes: trigger module 401, generation module 402, scheduling module 403, and control module 404.

[0059] Trigger module 401 is used to obtain data operation statistics information of the key-value storage engine in the storage service. When the data operation statistics information meets the preset trigger threshold, the data processing operation process is triggered. The generation module 402 is used to generate differentiated execution sequences for multiple storage services located on the same physical node that all meet the triggering conditions, so as to perform data processing operations on each storage service according to the execution sequence; The scheduling module 403 is used to coordinate and schedule data processing operations based on the node resource status and storage service distribution information obtained from the cluster management node. The control module 404 is used to control the corresponding storage service to perform data processing operations based on the coordination and scheduling results.

[0060] In some examples of this embodiment, the trigger module 401 is specifically configured to monitor and accumulate the number of deletion operations of the key-value storage engine, and / or monitor and accumulate the number of read and write operations of the key-value storage engine.

[0061] In some examples of this embodiment, the trigger module 401 is specifically configured to generate a trigger instruction to trigger a data cleanup operation when the number of deletion operations reaches a first dynamic threshold and / or the number of read / write operations reaches a second dynamic threshold.

[0062] In some examples of this embodiment, the generation module 402 is specifically configured to generate a random delay parameter for each storage service that meets the triggering conditions; convert the random delay parameter into a specific delay time according to a preset time mapping rule; and determine the specific time when each storage service starts the data processing operation based on the delay time.

[0063] In some examples of this embodiment, the generation module 402 is specifically configured to support dynamic configuration of time mapping rules and to adjust the length of the delay time according to the real-time load status of the node.

[0064] In some examples of this embodiment, the scheduling module 403 is specifically configured to collect the CPU utilization and memory utilization of each physical node by the cluster management node; and, in combination with the distributed topology of the storage service, use a preset scheduling algorithm to plan the execution order of data processing operations for the storage services on different physical nodes.

[0065] In some examples of this embodiment, the scheduling module 403 is specifically configured such that the preset scheduling algorithm ensures that the start time interval of any two data processing operations on the same physical node is not less than the preset minimum time window.

[0066] For a description of the features in the embodiment corresponding to the performance optimization device for the storage service, please refer to the relevant description in the embodiment corresponding to the performance optimization method for the storage service, which will not be repeated here.

[0067] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the performance optimization method for storage services.

[0068] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the performance optimization method for the storage service.

[0069] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0070] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described performance optimization method embodiments for storage services.

[0071] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described performance optimization method embodiments for storage services.

[0072] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0073] The foregoing has provided a detailed description of a performance optimization method, apparatus, and electronic device for a storage service provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A performance optimization method for a storage service, characterized in that, include: Obtain data operation statistics from the key-value storage engine in the storage service. When the data operation statistics meet a preset trigger threshold, trigger the data processing operation process. For multiple storage services located on the same physical node that all meet the triggering conditions, generate a differentiated execution sequence to perform data processing operations on each storage service according to the execution sequence; Based on the node resource status obtained from the cluster management node and the distribution information of the storage service, the data processing operation is coordinated and scheduled. Based on the coordination and scheduling results, control the corresponding storage service to perform data processing operations.

2. The performance optimization method for storage services according to claim 1, characterized in that, The acquisition of data operation statistics information of the key-value storage engine in the storage service includes: Monitor and accumulate the number of deletion operations of the key-value storage engine, and / or monitor and accumulate the number of read and write operations of the key-value storage engine.

3. The performance optimization method for storage services according to any one of claims 1-2, characterized in that, The triggered data processing operation process includes: When the number of deletion operations reaches a first dynamic threshold, and / or the number of read / write operations reaches a second dynamic threshold, a trigger command is generated to trigger the data cleanup operation.

4. The performance optimization method for storage services according to claim 1, characterized in that, The above refers to multiple storage services located on the same physical node that all meet the triggering conditions, generating a differentiated execution sequence, including: A random delay parameter is generated for each storage service that meets the triggering condition; According to the preset time mapping rules, the random delay parameters are converted into specific delay times; The specific time at which each of the storage services initiates data processing operations is determined based on the aforementioned delay time.

5. The performance optimization method for storage services according to claim 4, characterized in that, The time mapping rules support dynamic configuration and can adjust the delay time according to the real-time load of the nodes.

6. The performance optimization method for storage services according to claim 1, characterized in that, The coordination and scheduling of data processing operations based on the node resource status obtained from the cluster management node and the distribution information of the storage service includes: The cluster management node collects the CPU utilization and memory utilization of each physical node; Based on the distributed topology of the storage service, a preset scheduling algorithm is used to plan the execution order of data processing operations for the storage service on different physical nodes.

7. The performance optimization method for storage services according to claim 6, characterized in that, The preset scheduling algorithm ensures that on the same physical node, the start time interval between any two data processing operations is not less than the preset minimum time window.

8. The performance optimization method for storage services according to claim 1, characterized in that, Also includes: During the data processing operation, the resource utilization of the system's central processing unit and memory is continuously monitored; If the resource utilization rate continues to exceed the preset safety threshold, the cluster management node will issue an instruction to dynamically postpone subsequent data processing operations that are yet to be triggered or executed.

9. A performance optimization device for a storage service, characterized in that, include: The triggering module is used to obtain data operation statistics information of the key-value storage engine in the storage service. When the data operation statistics information meets the preset trigger threshold, the data processing operation process is triggered. The generation module is used to generate differentiated execution sequences for multiple storage services located on the same physical node and all meeting the triggering conditions, so as to perform data processing operations on each storage service according to the execution sequence. The scheduling module is used to coordinate and schedule data processing operations based on the node resource status obtained from the cluster management node and the distribution information of the storage service. The control module is used to control the corresponding storage service to perform data processing operations based on the coordination and scheduling results.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the performance optimization method for the storage service as described in any one of claims 1 to 8.

Images