Resource adjustment method, electronic device, storage medium and program product

Abstract

The application discloses a resource adjustment method, an electronic device, a storage medium and a program product, relates to the technical field of distributed storage, and can dynamically adjust the weight according to the use condition, burst state and priority of real-time IO resources of a client, better meets differentiated IO demands of multiple clients, can be in time and effective response to stable low-delay demands or periodic burst demands, and has strong dynamic adaptability; the application can flexibly and reasonably allocate bandwidth according to real-time demands of each client by dynamically adjusting a resource allocation strategy, enables the system to be more efficient, and provides services for more clients; in a distributed object storage environment, under the condition of equivalent hardware resources, the optimized IO scheduling algorithm enables the system to be more efficient when processing client requests, reduces system response delay, improves data read-write speed, and thus improves the performance of the entire distributed storage system.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology, and in particular to a resource adjustment method, electronic device, storage medium, and program product. Background Technology

[0002] In practical applications, IO (Input / Output) request traffic is often uneven, and sudden surges in traffic may occur. For example, operations such as database backup and video transcoding generate a large number of IO requests, causing the system to be overloaded instantly. Related IO scheduling algorithms usually use fixed resource allocation strategies, which are difficult to cope with such sudden IO situations, and can easily lead to a decrease in system performance and an increase in response latency. Therefore, how to dynamically adjust resources to cope with sudden IO demands and ensure the stability and reliability of the system is an urgent problem to be solved. Summary of the Invention

[0003] This application provides a resource adjustment method, electronic device, storage medium, and program product to at least solve the problem in related technologies that resources cannot be dynamically adjusted to cope with sudden I / O demands.

[0004] This application provides a resource adjustment method, including:

[0005] Initialize the resource allocation parameters of the target client;

[0006] Obtain the target client's status parameters, and based on these parameters, determine whether the target client's state is a sudden state.

[0007] In response to the target client's state being a sudden state, determine the resource adjustment parameters for the target client;

[0008] The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0009] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the following steps of a resource adjustment method when executing the computer program:

[0010] Initialize the resource allocation parameters of the target client;

[0011] Obtain the target client's status parameters, and based on these parameters, determine whether the target client's state is a sudden state.

[0012] In response to the target client's state being a sudden state, determine the resource adjustment parameters for the target client;

[0013] The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the following steps of a resource adjustment method:

[0015] Initialize the resource allocation parameters of the target client;

[0016] Obtain the target client's status parameters, and based on these parameters, determine whether the target client's state is a sudden state.

[0017] In response to the target client's state being a sudden state, determine the resource adjustment parameters for the target client;

[0018] The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the following steps of a resource adjustment method:

[0020] Initialize the resource allocation parameters of the target client;

[0021] Obtain the target client's status parameters, and based on these parameters, determine whether the target client's state is a sudden state.

[0022] In response to the target client's state being a sudden state, determine the resource adjustment parameters for the target client;

[0023] The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0024] This application can dynamically adjust weights based on the real-time IO resource usage, burst states, and priorities of clients, better meeting the differentiated IO needs of multiple clients. Whether it is a stable low-latency demand or a periodic burst demand, it can provide a timely and effective response, demonstrating strong dynamic adaptability. By dynamically adjusting the resource allocation strategy, this application can flexibly and reasonably allocate bandwidth according to the real-time needs of each client, enabling the system to operate more efficiently and provide services to more clients. In a distributed object storage environment, under the same hardware resource conditions, the optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response latency, and improves data read and write speed, thereby enhancing the performance of the entire distributed storage system. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This application provides a schematic diagram of a distributed storage system architecture for a resource adjustment method in an embodiment of the present application.

[0027] Figure 2 This application provides an overall flowchart of a resource adjustment method.

[0028] Figure 3 This application provides another overall flowchart illustrating a resource adjustment method.

[0029] Figure 4 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0033] With the rapid development of information technology, the amount of data generated by various industries has exploded. From the massive user data of Internet companies and the large-scale experimental data of scientific research institutions to the transaction records of the financial industry, the scale of data has far exceeded the carrying capacity of traditional centralized storage systems. Centralized storage usually relies on a single storage device or a few storage nodes, which has poor scalability. When the amount of data continues to increase, performance bottlenecks are likely to occur, such as slow read and write speeds and insufficient storage capacity. Moreover, centralized storage has the risk of single point of failure. Once the storage device fails, the data access of the entire system will be severely affected, and it may even lead to business interruption.

[0034] The rise of distributed computing technology has provided an opportunity for the development of distributed storage systems. Distributed computing can significantly improve computing efficiency by decomposing computing tasks and distributing them to multiple computing nodes for parallel processing. However, distributed computing requires efficient data storage and access support to ensure that each computing node can quickly obtain the required data. Distributed storage systems can distribute data across multiple nodes, matching the distributed computing architecture and enabling parallel reading and writing of data, thereby meeting the high-performance requirements of distributed computing for data storage and access.

[0035] As the background technology indicates, in practical applications, IO request traffic is often uneven, and sudden surges in traffic may occur. For example, operations such as database backup and video transcoding generate a large number of IO requests, leading to instantaneous high system load. Related IO scheduling algorithms typically employ fixed resource allocation strategies, which are ill-suited to handle such sudden IO surges, easily causing system performance degradation and increased response latency. Therefore, it is necessary to design an IO scheduling algorithm that can dynamically adjust resource allocation to cope with sudden IO demands and ensure system stability and reliability. In a multi-user environment, it is essential to ensure that each user or application receives fair resource allocation to meet its Quality of Service (QoS) requirements. For instance, for latency-sensitive applications such as real-time monitoring systems and online games, it is crucial to ensure that their IO requests are processed promptly to guarantee system real-time performance and response speed. Conversely, for applications with high throughput requirements, such as big data analysis and file transfer, it is necessary to ensure sufficient bandwidth resources to improve data processing efficiency. The IO scheduling algorithm needs to balance fairness and QoS assurance, rationally allocating storage resources according to the needs of different users or applications.

[0036] In distributed storage systems, common I / O scheduling algorithms include the following. Taking the dmclock algorithm as an example, its main steps are as follows:

[0037] Parameter initialization: Assign a fixed weight, reserved bandwidth, and maximum bandwidth to each client. These parameters are usually not dynamically adjusted during system operation.

[0038] Request queuing: When a client's IO request arrives, it is placed into the appropriate queue.

[0039] Scheduling decision: Based on the client's weight and the requests in the queue, select requests from the queue for processing according to certain rules. For example, prioritize processing requests from clients with higher weights, or process requests according to the first-in, first-out principle.

[0040] Bandwidth allocation: When processing requests, allocate corresponding bandwidth resources to the client based on the client's reserved bandwidth and maximum bandwidth.

[0041] The disadvantages of related technologies include:

[0042] Lack of dynamic adaptability: After parameter initialization, the relevant algorithms cannot dynamically adjust resource allocation according to the client's real-time IO usage and sudden demand. For example, when a client suddenly has a large number of IO requests, since its reserved bandwidth and maximum bandwidth are fixed, it may not be able to obtain enough resources to process these requests in time, resulting in increased response latency. This makes it difficult to meet the needs of uneven IO request traffic and sudden high traffic in actual applications.

[0043] Low resource utilization: Due to the fixed nature of the relevant algorithm parameters, some clients may have idle resources while other clients are unable to meet their needs due to insufficient resources, resulting in resource waste and reducing the overall resource utilization of the system.

[0044] Insufficient fairness: In a multi-client environment, the relevant algorithms cannot guarantee that each client can obtain fair resource allocation at different stages. For example, some clients may occupy too many resources for a long time due to unreasonable weight settings, affecting the normal operation of other clients and failing to meet the service quality (QoS) requirements of different users or applications. For example, applications such as latency-sensitive real-time monitoring systems and online games, as well as applications with high throughput requirements such as big data analysis and file transfer, cannot obtain appropriate resource allocation through the relevant algorithms.

[0045] Weak ability to handle sudden I / O: When operations such as database backup and video transcoding generate a large number of I / O requests, the I / O scheduling algorithm based on the relevant fixed resource allocation strategy is difficult to cope with, which can easily lead to a decrease in system performance and an increase in response latency, and cannot guarantee the stability and reliability of the system under sudden I / O conditions.

[0046] To address the aforementioned technical problems, this application provides a resource adjustment method, electronic device, storage medium, and program product. By designing an I / O scheduling algorithm capable of dynamically adjusting resource allocation, it aims to cope with sudden I / O demands and ensure system stability and reliability. In a multi-user environment, it is necessary to ensure that each user or application receives fair resource allocation to meet its Quality of Service (QoS) requirements. For example, for latency-sensitive applications such as real-time monitoring systems and online games, it is necessary to ensure that I / O requests are processed promptly to guarantee system real-time performance and response speed. Conversely, for applications with high throughput requirements, such as big data analysis and file transfer, it is necessary to ensure sufficient bandwidth resources to improve data processing efficiency. The I / O scheduling algorithm needs to balance fairness and QoS assurance, rationally allocating storage resources according to the needs of different users or applications.

[0047] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] The resource adjustment method provided in this application can be applied to, for example... Figure 1 The distributed storage system shown is designed based on scalability and fault tolerance, eliminating performance bottlenecks such as single-point metadata. It employs a distributed, decentralized architecture, slicing data and distributing it as objects across multiple nodes, ensuring balanced load and capacity across all nodes. Data access and management are achieved through network connections, while providing a unified storage interface. Because of its decentralized architecture, the distributed storage system is highly scalable; the failure of a single storage server node does not render the entire distributed storage cluster unavailable. It supports dynamic addition and removal of storage server nodes and can automatically relocate data objects within the cluster to balance storage space utilization across storage servers. The distributed storage client initiates an object I / O request to the object storage gateway. The gateway distributes the request to a unified, self-controlled, and scalable distributed storage consistency management system using a scalable hash calculation algorithm. This management system stores the object's index information, object metadata, and data in a storage pool. In the storage pool, the client's data is sliced, with a default fixed storage size of 4MB per object. Objects are grouped, and the group ID (unique identifier) ​​is the remainder of the hash (object x) and the number of groups. Finally, the data is stored on the underlying NVMe SSD or HDD. The distributed storage system architecture includes multiple components, such as the object gateway service, the unified, self-controlled, and scalable distributed storage consistency management system, the storage pool, the controllable, scalable, and distributed data balancing algorithm, and the metadata and monitoring service cluster.

[0049] Specifically, the distributed storage system architecture includes the following key components:

[0050] (1) Object Gateway Service: Provides a unified storage interface for distributed storage clients to access the interface service.

[0051] (2) Unified, self-controlled and scalable distributed storage consistency management system: This component is the core component of the distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects. Each object has a unique identifier and related data. It is responsible for distributing the objects to various nodes of the storage cluster and provides data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access.

[0052] (3) Storage pool: The distributed storage cluster consists of multiple storage nodes. Each storage node can contain multiple hard disks or storage devices. The storage pool divides the storage resources to form different pools to meet different storage needs. The data storage pool is composed of HDDs (hard disk drives) to take cost into consideration. The metadata storage pool uses NVMe SSDs to ensure efficient metadata access operations. Data and metadata are stored in the form of objects (object storage devices, abbreviated as OSDs).

[0053] (4) Controllable, scalable, distributed data balancing algorithm: This component distributes data objects evenly across the nodes of the storage cluster to avoid data hotspots and improve system performance.

[0054] (5) Monitoring Service Cluster: The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster. Its technical principle includes maintaining the status information, configuration information, health status, etc. of the cluster to ensure the consistency and availability of the cluster.

[0055] The improvements in this application are mainly applied to the OSD (Object Storage Device, which stores data as objects on the physical disks of each node in the cluster) module in the distributed storage system. The OSD is the core component responsible for storing data in the distributed storage system. It directly handles the client's IO requests. In the OSD module, the IO scheduling algorithm has the aforementioned shortcomings. This application improves the IO scheduling algorithm in the OSD module so that it can dynamically adjust resource allocation according to the client's real-time IO usage and sudden demand, thereby improving the system's performance and resource utilization.

[0056] like Figure 2 As shown, embodiments of this application provide a resource adjustment method, which is applied to... Figure 1Taking a distributed storage system as an example, the following steps are included:

[0057] S1: Initialize the resource allocation parameters for the target client.

[0058] It should be noted that the target client refers to one of the multiple clients corresponding to the distributed storage system, and the resource allocation parameters include the reserved bandwidth r. c Burst bandwidth b c Total IO quota IOQ c and priority P c Simultaneously initialize the system's total available bandwidth C, time series period T, burst weight adjustment coefficient α, and maximum burst duration T. burst and minimum bandwidth threshold B min And the cumulative I / O usage (IOU) for each client. c Initialize to 0. Initialization means assigning default values ​​to the variables, i.e., assigning values ​​to the parameters mentioned above. The time series period T is in seconds; it's a unit for measuring client resource usage over a period of time, such as 300 seconds, 600 seconds, etc. IOQ c This represents the total I / O quota for client c within a time series period T, reflecting the maximum amount of I / O allowed for the client during that period. c The reserved bandwidth for client c, measured in IOPS (Input / Output Operations Per Second), is the basic bandwidth that the client can always guarantee. c This refers to the burst bandwidth of client c, also measured in IOPS, used to meet the bursty IO demands of the client at specific times. (IOU) c ω refers to the cumulative I / O usage of client c up to time t, recording the client's I / O usage from the beginning to the current moment; c (t) represents the dynamic weight of client c at time t, which changes in real time based on the client's IO usage. C is the total available bandwidth of the system, measured in IOPS, representing the maximum IO bandwidth that the system can provide at a given moment. P c This indicates the priority of client c, with a value ranging from 1 to 5. The larger the number, the higher the priority. It can be preset according to factors such as the importance of the client's business.

[0059] S2: Obtain the state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0060] It should be noted that the status parameter refers to the client's request rate at different time points. The required bandwidth for the request can be determined by the request rate and the corresponding request size. The burst status refers to the occurrence of a burst of IO requests, that is, a sudden high traffic situation.

[0061] S3: In response to the target client's state being a burst state, determine the target client's resource adjustment parameters.

[0062] It should be noted that resource adjustment parameters generally include dynamic weights and allocated bandwidth. By adjusting the allocated bandwidth, the request processing needs of the target client can be met.

[0063] S4: Adjust the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocate resources to the target client according to the adjusted resource allocation parameters.

[0064] It should be noted that the allocated resources generally refer to the OSD having available resources, which are allocated to the target client to handle the target client's input and output requests.

[0065] In the above implementation, the weights can be dynamically adjusted according to the real-time IO resource usage, burst states, and priorities of the clients, which can better meet the differentiated IO needs of multiple clients. Whether it is a stable low-latency demand or a periodic burst demand, it can get a timely and effective response, with strong dynamic adaptability. This application can flexibly and reasonably allocate bandwidth according to the real-time needs of each client by dynamically adjusting the resource allocation strategy, so that the system can run more efficiently and provide services to more clients. In a distributed object storage environment, under the same hardware resource conditions, the optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response latency, improves data read and write speed, and thus improves the performance of the entire distributed storage system.

[0066] In some specific implementations, the resource allocation parameters for initializing the target client include:

[0067] Determine whether the current time series period is the initial time series period;

[0068] In response to the current time series period being the initial time series period, the initial resource allocation parameters are defined as resource allocation parameters;

[0069] In response to the fact that the current time series period is not the initial time series period, the resource allocation parameters of the previous time series period are adjusted based on the resource usage of the previous time series period, and the adjusted resource allocation parameters are defined as resource allocation parameters.

[0070] Specifically, the initial time series period refers to the first time series period when the system just starts running. The initial resource allocation parameters refer to the initial default values ​​of the resource allocation parameters. When the current time series period is the first time series period, the initial default values ​​of the resource allocation parameters are defined as the resource allocation parameters. When the current time series period is not the first time series period, the resource allocation parameters are adjusted based on the resource usage of the previous time series period, and the adjusted resource allocation parameters are defined as the resource allocation parameters for the current time series period. The initialization process involves determining the reserved bandwidth r for each client c based on the client's needs and the system resource situation. c Burst bandwidth b c Total IO quota IOQ c and priority P c For example, for clients with high latency requirements, a higher reserved bandwidth can be allocated. c and higher priority P c For clients experiencing sudden surges in demand, increase the burst bandwidth (b) as needed. c Simultaneously, determine the total available bandwidth C, time series period T, burst weight adjustment coefficient α, and maximum burst duration T of the system. burst and minimum bandwidth threshold B min The total available bandwidth C depends on the hardware configuration of the storage system, and the time series period T can be set according to actual business needs; the burst weight adjustment coefficient α can be set to 0.5, and the maximum burst duration T... burst It can be set to 30 seconds, minimum bandwidth threshold B min This can be configured according to the actual situation of the system, setting the cumulative I / O usage (IOU) for each client. c (0) is initialized to 0 to prepare for subsequent IO usage statistics.

[0071] In the above implementation, the initial resource allocation parameters for different time series periods are determined, thereby enabling more accurate determination of resource allocation values ​​and improving resource utilization and the dynamic adaptability of the system.

[0072] In some specific implementations, such as Figure 3 As shown, before obtaining the target client's status parameters, the resource adjustment method also includes:

[0073] The system detects whether the target client has issued an input / output request. If so, it receives the input / output request from the target client. Here, the input / output request refers to an IO request. When the client initiates an IO request, it sends the IO request to the MDS (Meta Data Server) of the distributed storage system. The MDS is responsible for managing the metadata of the file system. After receiving the request, the MDS determines the OSD set corresponding to the request based on the file path and metadata information of the request, and forwards the request to the corresponding OSD. After receiving the IO request from the MDS (which is a component of SQL Server and is mainly used to manage the master data integration of enterprise data) through the OSD, it puts it into the global request queue. The global request queue is responsible for storing the requests to be processed. It ensures that each request is processed in order through the FIFO (First Input First Output) mechanism. When multiple requests arrive concurrently, the queue can effectively manage the order of these requests and avoid processing chaos.

[0074] In response to receiving input / output requests from the target client, the system obtains the target client's status parameters and records the current time. The status parameters include the request rate and size of the I / O request.

[0075] In the above implementation, the target client's status parameters are obtained by detecting whether an input / output request is received, in order to subsequently determine the target client's status.

[0076] In some specific implementations, determining whether the state of the target client is a sudden state based on the target client's state parameters includes:

[0077] Obtain the bandwidth required for the corresponding request rate of the input and output requests sent by the target client and the reserved bandwidth of the target client. The required bandwidth can be calculated based on the request rate and request size. The calculation method is a common method, and the specific calculation process will not be described here.

[0078] In response to a reserved bandwidth exceeding a preset multiple, the target client's state is determined to be in a burst state, and a burst state flag is set for the target client. The preset multiple can be set according to actual needs, such as 2 times, and the burst state flag is S. c (t)=1, specifically, monitor the IO request rate of each client c in real time, and determine whether the bandwidth required for the IO request rate of client c exceeds its reserved bandwidth r. c Twice as much; if it exceeds this, then S will be... c (t) is set to 1, and the start time t of the sudden state is recorded. start ;

[0079] In response to a reserved bandwidth that is less than or equal to a preset multiple, the system determines that the target client's state is not a bursty state and sets a non-burst state flag for the target client, where the non-burst state flag is S. c (t)=0, where the bandwidth required for the IO request rate of client c does not exceed its reserved bandwidth r. c If it is twice the value of S, then S will be... c (t) is set to 0.

[0080] In some specific real-time scenarios, resource adjustment methods also include:

[0081] The system retrieves clients that exhibit abrupt changes within multiple time series periods, and marks these clients based on time nodes. The number of time series periods can be set according to actual needs.

[0082] When the number of occurrences of the same time node exceeds a preset threshold, the time node is marked once. The duration of the state change of the client at the time node in different time series periods is determined. The preset threshold can be set according to actual needs.

[0083] When the number of time series periods corresponding to the duration of the mutation state being within a preset range is greater than a preset threshold, the time node is marked a second time. The preset range can be set according to actual needs, i.e., the duration of the mutation state is greater than a preset value.

[0084] When the current time series period reaches the time node that has been marked twice, the state of the target client at that time node is defined as being in a burst state.

[0085] In the above implementation, by marking the time nodes where multiple sudden states occur, the target client is directly defined as being in a sudden state when the next time series period reaches that time node, thereby improving the efficiency of resource adjustment.

[0086] In some specific implementations, after determining that the target client is in a burst state and setting a burst state flag for the target client, the resource adjustment method further includes:

[0087] Obtain the start time of the sudden state, i.e., t. start And based on the start time, determine the duration of the emergency;

[0088] If the duration of a sudden event exceeds the preset sudden event time, the target client's state is defined as a non-sudden event, and the target client's state flag is reset to the non-sudden event flag. The preset sudden event time is the maximum sudden event duration T. startThe specific value of its maximum emergency duration can be set according to actual needs, that is, the duration of the emergency exceeds the maximum emergency duration T. start Then S c (t) is reset to 0.

[0089] In the above implementation, the state of the target client is judged to determine whether it is a sudden state, so as to adjust the resources of the target client under the sudden state, so that the client can run stably, improve the dynamic adaptability of the system, and when the sudden state lasts for a long time, the flag is reset to 0 to accurately measure the client's resource demand, so as to avoid a certain client occupying too many resources for a long time, thereby improving resource utilization.

[0090] In some specific implementations, in response to the target client's state being a sudden state, the resource adjustment parameters for determining the target client include:

[0091] Based on the resource usage of the target client, determine the dynamic weight of the target client for determining bandwidth allocation;

[0092] The allocated bandwidth for the target client is determined based on dynamic weights.

[0093] In some specific implementations, determining the dynamic weights for determining bandwidth allocation for the target client based on the target client's resource usage includes:

[0094] Get the resource usage (IOU) of the target client from the start time to the current time within a time series period. c (t), and the reserved bandwidth r of the target client. c Wherein, the time period from the start time to the current time is t;

[0095] Based on resource usage IOU c (t) and the reserved bandwidth r of the target client c The target client is identified to determine the dynamic weights used to allocate bandwidth.

[0096] In some specific implementations, the target client is used to determine the dynamic weight of the allocated bandwidth, that is, the dynamic weight ω of the target client at time t. c (t), including:

[0097] ;

[0098] Where, ω c (t) represents the dynamic weight, S c (t) represents the status flag of the target client c, α represents the preset burst weight adjustment coefficient, and P c Indicates the priority of the target client c, t represents time, and b represents the priority of the target client c.c IOQ represents burst bandwidth, T represents the time series period, and IOQ represents burst bandwidth. c (t) represents the total input / output quota within the time series period, where the preset burst weight adjustment coefficient is used to increase the dynamic weight of a client when a burst I / O occurs, so as to allocate more resources to it. For example, when client c experiences a burst I / O (S c When (t)=1), its dynamic weight will increase accordingly, and this will be combined with the client's priority P. c Further adjust its priority in resource allocation.

[0099] In some specific implementations, determining the allocated bandwidth for the target client based on dynamic weights includes:

[0100] Based on the bandwidth calculation function and dynamic weights, the bandwidth allocated to the target client during the arrival time period t is determined. The bandwidth calculation function includes:

[0101] ;

[0102] Among them, B c (t) represents the allocated bandwidth, n represents the number of clients, and ω j (t) represents the weight value of the j-th client, and C represents the total available bandwidth of the system.

[0103] For example, for each client c, ω is calculated according to the dynamic weight calculation formula. c (t), for example, if the IOU of client c c (t) is less than or equal to r c *t and S c (t)=1, P c =3, then use Calculate the weights and the bandwidth allocated to each client, B, according to the bandwidth allocation formula. c For example, in a system with 3 clients, client 1 has ω1(t) = 0.6, client 2 has ω2(t) = 0.3, and client 3 has ω3(t) = 0.1. The total available bandwidth of the system is C = 1000 IOPS. Then, the bandwidth allocated to client 1 is... If the calculated Less than the minimum bandwidth threshold B min Then set B1(t) as B min And so on, without further explanation.

[0104] In some specific implementations, adjusting the resource allocation parameters of the target client based on the target client's resource adjustment parameters includes:

[0105] In response to the target client's allocated bandwidth being greater than or equal to a preset bandwidth threshold, the allocated bandwidth is defined as the target client's resource adjustment parameter in order to adjust the target client's resource allocation parameters;

[0106] In response to the target client's allocated bandwidth being less than a preset bandwidth threshold, the preset bandwidth threshold is defined as the target client's resource adjustment parameter, so as to adjust the target client's resource allocation parameters.

[0107] The preset bandwidth threshold can be set according to actual needs, and is generally the minimum bandwidth threshold.

[0108] In the above implementation, the dynamic weight of each client is determined by a weight calculation function, and the total available bandwidth of the allocation system is determined based on the dynamic weight to ensure the fairness and dynamism of resource allocation. Simultaneously, to prevent a client from being unable to process requests properly due to insufficient allocated bandwidth, a minimum bandwidth threshold B is set. min If the calculated B c (t) is less than B min Then set the dynamic weight to B. min This avoids insufficient bandwidth allocation to the client affecting request processing and improves the overall performance of the system.

[0109] In some specific implementations, allocating resources to the target client according to the adjusted resource allocation parameters includes:

[0110] Based on the allocated bandwidth, the input and output requests of the target client are obtained from the global request queue and placed into the pending queue.

[0111] When available resources are available, retrieve the target client's input / output requests from the pending queue for processing and update the target client's resource usage.

[0112] Specifically, when an IO request arrives, the current time t is recorded. First, burst state detection is performed to determine the S for each client. c The value of (t) is then calculated, and for each client c, ω is calculated according to the dynamic weight calculation formula described above. c (t), and then calculate the bandwidth B allocated to each client according to the bandwidth allocation formula. c (t) retrieves each client's I / O request from the global request queue according to the allocated bandwidth and places it into the corresponding waiting queue. When the OSD has idle resources, it retrieves the request from the waiting queue for processing. During the request processing, the IOU is updated. cThe OSD is responsible for handling I / O requests in real time, dynamically adjusting resource allocation based on the actual situation of the client. If the client's I / O request rate changes during processing, causing a sudden change in the state, the OSD will update the resource allocation in a timely manner. c The values ​​of (t) and C are calculated, and the dynamic weights and bandwidth are recalculated.

[0113] In the above implementation, dynamic weights and bandwidth allocation formulas are used, combined with client priority, to ensure that each client receives fair resource allocation at different stages. This avoids the problem of uneven resource allocation caused by unreasonable weight settings in related algorithms, and guarantees the interests of each client. For latency-sensitive real-time monitoring systems and online games, as well as big data analysis and file transfer applications with high throughput requirements, resources can be allocated reasonably according to their needs, thereby improving the overall performance of the system.

[0114] In some specific implementations, the resource adjustment method further includes:

[0115] In response to the end of the current time series period, obtain the resource usage of the target client in the current time series period;

[0116] In response to the target client's resource usage exceeding the total input / output quota, the reserved bandwidth and burst bandwidth for the next time series period are reduced and adjusted, and the target client's resource usage is reset.

[0117] In response to the target client's resource usage being less than or equal to the total input / output quota, the reserved bandwidth and burst bandwidth for the next time series period are increased and adjusted, and the target client's resource usage is reset.

[0118] Specifically, the OSD stores data on the local disk based on the request type (e.g., write request). During storage, it follows the data redundancy and consistency strategies of the distributed storage system to ensure data reliability. Once the request is processed, the OSD returns the result to the MDS, which then returns the result to the client. After receiving the response, the client completes the I / O operation. At the end of each time series period T, the IOU of each client is checked. c Does it exceed IOQ? c If the limit is exceeded, the reserved bandwidth and burst bandwidth for that client will be adjusted in the next billing cycle, meaning the reserved bandwidth for that client will be reduced in the next billing cycle. and burst bandwidth For example, reserved bandwidth can be reduced by 10%, and burst bandwidth by 20% to penalize clients that overuse resources. At the same time, the cumulative I / O usage (IOU) of each client can be reset. c=0, start a new time series cycle, thereby ensuring that clients use resources reasonably within a time series cycle, maintaining the fairness and stability of the system. If the limit is not exceeded, the reserved bandwidth and burst bandwidth of the client can be appropriately increased as needed, such as increasing the reserved bandwidth by 5% and the burst bandwidth by 10%, to reward clients that use resources reasonably. At the same time, reset the cumulative IO usage (IOQ) of each client. c =0, and start a new time series cycle to ensure that resource usage statistics restart.

[0119] In the above implementation, by comparing the resource usage and total input / output quota of the previous time series period, the reserved bandwidth and burst bandwidth of the next time series period are adjusted, thereby improving the accuracy of resource allocation and further enhancing the overall performance of the system.

[0120] The above resource adjustment method includes: initializing the resource allocation parameters of the target client; obtaining the status parameters of the target client, and determining whether the target client's status is a burst state based on the status parameters; in response to the target client's status being a burst state, determining the resource adjustment parameters of the target client; adjusting the resource allocation parameters of the target client based on the resource adjustment parameters, and allocating resources to the target client according to the adjusted resource allocation parameters. The beneficial effects of the above method are:

[0121] (1) Strong dynamic adaptability: The algorithm can dynamically adjust the weight according to the client's real-time IO usage, sudden state and priority. In actual application scenarios, when the client has a sudden IO demand, such as database backup or video transcoding generating a large number of requests, the system can promptly detect and increase its dynamic weight, and allocate more resources to the client. Compared with the scheduling algorithm with fixed parameters, this application can better meet the differentiated IO needs of multiple clients. Whether it is a stable low-latency demand or a periodic sudden demand, it can get a timely and effective response.

[0122] (2) High resource utilization: By dynamically adjusting the resource allocation strategy, the system can make full use of the remaining bandwidth. Related algorithms may lead to some client resources being idle. However, this application can allocate these idle resources to clients that need them, reducing bandwidth waste. When multiple clients access the system concurrently, bandwidth can be flexibly allocated according to the real-time needs of each client, enabling the system to run more efficiently and provide services to more clients.

[0123] (3) Fairness is guaranteed: By using dynamic weights and bandwidth allocation formulas, and combining them with client priority, it is ensured that each client can obtain fair resource allocation at different stages, avoiding the problem of uneven resource allocation caused by unreasonable weight settings in related algorithms, and guaranteeing the interests of each client. For latency-sensitive real-time monitoring systems and online games, as well as big data analysis and file transfer applications with high throughput requirements, resources can be reasonably allocated according to their needs.

[0124] (4) Improved system performance: In a distributed object storage environment, under the same hardware resource conditions, the overhead of distributed metadata occupying NVMe SSD space can be reduced. The optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response latency, and improves data read and write speed, thereby improving the performance of the entire distributed storage system.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0126] It should be understood that, although Figures 1-3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0127] In one embodiment, an electronic device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the electronic device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a resource adjustment method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0128] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0129] Embodiments of this application provide an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in the resource adjustment method embodiments, including:

[0130] S1: Initialize the resource allocation parameters for the target client.

[0131] S2: Obtain the state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0132] S3: In response to the target client's state being a burst state, determine the target client's resource adjustment parameters.

[0133] S4: Adjust the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocate resources to the target client according to the adjusted resource allocation parameters.

[0134] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the resource adjustment method embodiments at runtime, including:

[0135] S1: Initialize the resource allocation parameters for the target client.

[0136] S2: Obtain the state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0137] S3: In response to the target client's state being a burst state, determine the target client's resource adjustment parameters.

[0138] S4: Adjust the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocate resources to the target client according to the adjusted resource allocation parameters.

[0139] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0140] Embodiments of this application also provide a computer program product, which includes a computer program. When executed by a processor, the computer program implements the steps in the resource adjustment method embodiments, including:

[0141] S1: Initialize the resource allocation parameters for the target client.

[0142] S2: Obtain the state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0143] S3: In response to the target client's state being a burst state, determine the target client's resource adjustment parameters.

[0144] S4: Adjust the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocate resources to the target client according to the adjusted resource allocation parameters.

[0145] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps in the resource adjustment method embodiments, including:

[0146] S1: Initialize the resource allocation parameters for the target client.

[0147] S2: Obtain the state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0148] S3: In response to the target client's state being a burst state, determine the target client's resource adjustment parameters.

[0149] S4: Adjust the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocate resources to the target client according to the adjusted resource allocation parameters.

[0150] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0151] The resource adjustment method, apparatus, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A resource adjustment method, characterized in that, The method includes: Initialize the resource allocation parameters of the target client; Obtain the status parameters of the target client, and based on the status parameters of the target client, determine whether the status of the target client is a sudden state; In response to the target client's state being a sudden state, the resource adjustment parameters for the target client are determined; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters; In response to the target client's state being a sudden state, the resource adjustment parameters for the target client are determined as follows: Based on the resource usage of the target client, determine the dynamic weight of the target client for determining bandwidth allocation, including: Obtain the resource usage (IOU) of the target client from the start time to the current time within the time series period. c (t), and the reserved bandwidth r of the target client. c ; Based on the resource usage IOU c (t) and the reserved bandwidth r of the target client c Determining the dynamic weights used by the target client to determine bandwidth allocation includes: ； Where, ω c (t) represents the dynamic weight, S c (t) represents the status flag of the target client c, α represents the preset burst weight adjustment coefficient, and P c Indicates the priority of the target client c, t represents time, and b represents the priority of the target client c. c IOQ represents burst bandwidth, T represents the time series period, and IOQ represents burst bandwidth. c (t) represents the total input / output quota within the time series period; The allocated bandwidth for the target client is determined based on the dynamic weights.

2. The resource adjustment method according to claim 1, characterized in that, The initialization parameters for the target client's resource allocation include: Determine whether the current time series period is the initial time series period; In response to the current time series period being the initial time series period, the initial resource allocation parameter is defined as the resource allocation parameter; In response to the fact that the current time series period is not the initial time series period, the resource allocation parameters of the previous time series period are adjusted based on the resource usage of the previous time series period, and the adjusted resource allocation parameters are defined as the resource allocation parameters.

3. The resource adjustment method according to claim 1, characterized in that, Before obtaining the state parameters of the target client, the method further includes: Detect whether the target client has issued an input / output request; if so, receive the input / output request issued by the target client. In response to receiving an input / output request from the target client, the system obtains the target client's status parameters and records the current time.

4. The resource adjustment method according to claim 1, characterized in that, Determining whether the state of the target client is a sudden state based on the target client's state parameters includes: Obtain the bandwidth required for the request rate corresponding to the input / output requests sent by the target client and the reserved bandwidth of the target client; In response to the reserved bandwidth being greater than a preset multiple, the state of the target client is determined to be a burst state, and a burst state flag is set for the target client; In response to the reserved bandwidth being less than or equal to a preset multiple of the required bandwidth, it is determined that the state of the target client is not a burst state, and a non-burst state flag is set for the target client.

5. The resource adjustment method according to claim 4, characterized in that, After determining that the target client is in a sudden state and setting a sudden state flag for the target client, the method further includes: Obtain the start time of the sudden state, and determine the duration of the sudden state based on the start time; In response to the fact that the duration of the sudden state is greater than the preset sudden state time, the state of the target client is defined as a non-sudden state, and the state identifier of the target client is reset to the non-sudden state identifier.

6. The resource adjustment method according to claim 1, characterized in that, The bandwidth allocation for the target client is determined based on the dynamic weights, including: Based on the bandwidth calculation function and the dynamic weight, the allocated bandwidth for the target client is determined. The bandwidth calculation function includes: ； Among them, B c (t) represents the allocated bandwidth, n represents the number of clients, and ω j (t) represents the weight value of the j-th client, and C represents the total available bandwidth of the system.

7. The resource adjustment method according to claim 1, characterized in that, Adjusting the resource allocation parameters of the target client based on the target client's resource adjustment parameters includes: In response to the target client's allocated bandwidth being greater than or equal to a preset bandwidth threshold, the allocated bandwidth is defined as the target client's resource adjustment parameter, so as to adjust the target client's resource allocation parameter; In response to the target client's allocated bandwidth being less than a preset bandwidth threshold, the preset bandwidth threshold is defined as the target client's resource adjustment parameter, so as to adjust the target client's resource allocation parameter.

8. The resource adjustment method according to claim 7, characterized in that, Allocating resources to the target client according to the adjusted resource allocation parameters includes: Based on the allocated bandwidth, the input / output requests of the target client are obtained from the global request queue, and the input / output requests of the target client are placed into the processing queue; When there are available resources, the input / output requests of the target client are retrieved from the queue to be processed and the resource usage of the target client is updated.

9. The resource adjustment method according to claim 1, characterized in that, The method further includes: In response to the end of the current time series period, obtain the resource usage of the target client in the current time series period; In response to the target client's resource usage exceeding the total input / output quota, the reserved bandwidth and burst bandwidth for the next time series period are reduced and adjusted, and the target client's resource usage is reset. In response to the target client's resource usage being less than or equal to the total input / output quota, the reserved bandwidth and burst bandwidth for the next time series period are increased and adjusted, and the target client's resource usage is reset.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the resource adjustment method as described in any one of claims 1 to 9 when executing the computer program.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the resource adjustment method as described in any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the resource adjustment method as described in any one of claims 1 to 9.

Images