Resource adjustment method, electronic device, storage medium and program product

Abstract

The invention discloses a resource adjustment method, electronic equipment, a storage medium and a program product, and relates to the technical field of distributed storage, the weight can be dynamically adjusted according to the use condition, the emergency state and the priority of real-time IO resources of clients, the differentiated IO requirements of multiple clients can be better met, and the user experience is improved. A stable low-delay demand or a periodic burst demand can be timely and effectively responded, and the dynamic adaptability is high; by dynamically adjusting the resource allocation strategy, the bandwidth can be flexibly and reasonably allocated according to the real-time demand of each client, so that the system can operate more efficiently, and services are provided for more clients; in a distributed object storage environment and under the same hardware resource condition, the optimized IO scheduling algorithm enables the system to be more efficient when processing a client request, system response delay is reduced, the data read-write speed is increased, and therefore the performance of the whole distributed storage system is improved.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a resource adjustment method, electronic device, storage medium, and program product. Background Art

[0002] In practical applications, the traffic of IO (Input / Output) requests is often uneven, and sudden high traffic situations may occur. For example, in operations such as database backup and video transcoding, a large number of IO requests are generated, resulting in instantaneous excessive load on the system. Related IO scheduling algorithms usually adopt fixed resource allocation strategies, which are difficult to cope with such sudden IO situations, and can easily cause system performance degradation and increased response delays. Therefore, how to dynamically adjust resources to cope with sudden IO demands and ensure system stability and reliability is an urgent problem that needs to be solved. Summary of the Invention

[0003] The present application provides a resource adjustment method, an electronic device, a storage medium, and a program product to at least solve the problem in the related art that resources cannot be dynamically adjusted to cope with sudden IO demands.

[0004] This application provides a resource adjustment method, including: Initialize resource allocation parameters of the target client; Obtaining a state parameter of a target client, and determining whether the state of the target client is an emergency state based on the state parameter of the target client; In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0005] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the following steps of the resource adjustment method when executing the computer program: Initialize resource allocation parameters of the target client; Obtaining a state parameter of a target client, and determining whether the state of the target client is an emergency state based on the state parameter of the target client; In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0006] The present application also provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed by a processor, the following steps of the resource adjustment method are implemented: Initialize resource allocation parameters of the target client; Obtaining a state parameter of a target client, and determining whether the state of the target client is an emergency state based on the state parameter of the target client; In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0007] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps of the resource adjustment method: Initialize resource allocation parameters of the target client; Obtaining a state parameter of a target client, and determining whether the state of the target client is an emergency state based on the state parameter of the target client; In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

[0008] This application can dynamically adjust the weight according to the client's real-time IO resource usage, burst status and priority, and can better meet the differentiated IO needs of multiple clients. Whether it is a stable low-latency demand or a periodic burst demand, it can get a timely and effective response and has strong dynamic adaptability. This application can flexibly and reasonably allocate bandwidth according to the real-time needs of each client by dynamically adjusting the resource allocation strategy, so that the system can run more efficiently and provide services to more clients. In a distributed object storage environment, under the same hardware resource conditions, the optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response delay, and improves data reading and writing speed, thereby improving the performance of the entire distributed storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 A schematic diagram of a distributed storage system architecture providing a resource adjustment method according to an embodiment of the present application; Figure 2 A schematic diagram of the overall process of a resource adjustment method is provided for an embodiment of the present application; Figure 3 Another overall flow chart of a resource adjustment method is provided for an embodiment of the present application; Figure 4 FIG. 1 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0011] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative work are within the scope of protection of this application.

[0012] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0013] It should be noted that the terms "S1", "S2", etc. are used only for the purpose of describing the steps and do not specifically refer to the order or sequence, nor are they used to limit this application. They are merely for the convenience of describing the method of this application and should not be understood as indicating the order of the steps. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that ordinary technicians in this field can implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0014] With the rapid development of information technology, the amount of data generated by various industries has shown explosive growth. From the massive user data of Internet companies, the large-scale experimental data of scientific research institutions, to the transaction records of the financial industry, the data scale has far exceeded the carrying capacity of traditional centralized storage systems. Centralized storage usually relies on a single storage device or a few storage nodes, and its scalability is poor. When the amount of data continues to increase, performance bottlenecks such as slow read and write speeds and insufficient storage capacity are prone to occur. In addition, centralized storage has the risk of single point failure. Once a storage device fails, data access of the entire system will be seriously affected, and even business interruption will occur.

[0015] The rise of distributed computing technology has provided an opportunity for the development of distributed storage systems. Distributed computing can significantly improve computing efficiency by decomposing computing tasks and assigning them to multiple computing nodes for parallel processing. However, distributed computing requires efficient data storage and access support to ensure that each computing node can quickly obtain the required data. Distributed storage systems can store data in a dispersed manner across multiple nodes, matching the distributed computing architecture and realizing parallel reading and writing of data, thereby meeting the high-performance requirements of distributed computing for data storage and access.

[0016] As can be seen from the background technology, in practical applications, the traffic of IO requests is often uneven, and sudden high traffic situations may occur. For example, in operations such as database backup and video transcoding, a large number of IO requests are generated, resulting in an instantaneous overload of the system. Related IO scheduling algorithms usually adopt a fixed resource allocation strategy, which is difficult to cope with such sudden IO situations and easily causes system performance degradation and increased response delay. Therefore, it is necessary to design an IO scheduling algorithm that can dynamically adjust resource allocation to cope with sudden IO demands and ensure system stability and reliability. In a multi-user environment, it is necessary to ensure that each user or application receives fair resource allocation to meet its quality of service (QoS) requirements. For example, for some delay-sensitive applications, such as real-time monitoring systems and online games, it is necessary to ensure that their IO requests can be processed in a timely manner to ensure the real-time and responsiveness of the system. For some applications with high throughput requirements, such as big data analysis and file transfer, it is necessary to ensure that they have sufficient bandwidth resources to improve data processing efficiency. IO scheduling algorithms need to strike a balance between fairness and QoS assurance and reasonably allocate storage resources according to the needs of different users or applications.

[0017] In distributed storage systems, common IO scheduling algorithms include the following. Taking the dmclock algorithm as an example, its main steps are as follows: Parameter initialization: Each client is assigned a fixed weight, reserved bandwidth, and maximum bandwidth. These parameters are usually not dynamically adjusted during system operation.

[0018] Request queuing: When a client's IO request arrives, it is placed in the corresponding queue.

[0019] Scheduling decision: Based on the client weight and the number of requests in the queue, requests are selected from the queue for processing according to certain rules. For example, requests from clients with high weights are prioritized, or requests are processed according to the first-in-first-out principle.

[0020] Bandwidth allocation: When processing a request, appropriate bandwidth resources are allocated to the client based on the client's reserved bandwidth and maximum bandwidth.

[0021] Disadvantages of related technologies include: Lack of dynamic adaptability: After parameter initialization, the relevant algorithms cannot dynamically adjust resource allocation based on the client's real-time IO usage and sudden demands. For example, when a client suddenly receives a large number of IO requests, it may not be able to obtain sufficient resources in time to process these requests because its reserved bandwidth and maximum bandwidth are fixed, resulting in increased response delays and difficulty in meeting the uneven IO request traffic and sudden high traffic conditions in actual applications.

[0022] Low resource utilization: Due to the fixed nature of relevant algorithm parameters, some clients may have idle resources while other clients cannot meet their needs due to insufficient resources, resulting in a waste of resources and reducing the overall resource utilization of the system.

[0023] Insufficient fairness: In a multi-client environment, relevant algorithms cannot guarantee fair resource allocation for each client at different stages. For example, some clients may occupy excessive resources for a long time due to unreasonable weight settings, affecting the normal operation of other clients and failing to meet the quality of service (QoS) requirements of different users or applications. For example, applications such as real-time monitoring systems and online games that are sensitive to latency, as well as applications such as big data analysis and file transfer that have high throughput requirements, cannot obtain appropriate resource allocation through relevant algorithms.

[0024] Weak ability to cope with sudden IO: When operations such as database backup and video transcoding generate a large number of IO requests, the IO scheduling algorithm of the relevant fixed resource allocation strategy is difficult to cope with, which can easily cause system performance to degrade and response delay to increase, and cannot guarantee the stability and reliability of the system in the case of sudden IO.

[0025] To solve the above technical problems, the present application provides a resource adjustment method, an electronic device, a storage medium, and a program product. By designing an IO scheduling algorithm that can dynamically adjust resource allocation to cope with sudden IO demands and ensure the stability and reliability of the system, in a multi-user environment, it is necessary to ensure that each user or application can obtain fair resource allocation to meet its quality of service (QoS) requirements. For example, for some delay-sensitive applications, such as real-time monitoring systems and online games, it is necessary to ensure that their IO requests can be processed in a timely manner to ensure the real-time performance and response speed of the system; and for some applications with high throughput requirements, such as big data analysis and file transfer, it is necessary to ensure that they can obtain sufficient bandwidth resources to improve data processing efficiency. The IO scheduling algorithm needs to strike a balance between fairness and service quality assurance, and reasonably allocate storage resources according to the needs of different users or applications.

[0026] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] The resource adjustment method provided in this application can be applied to Figure 1 The distributed storage system shown in the figure is designed based on scalability and fault tolerance, eliminating performance bottlenecks such as single-point metadata. It adopts a distributed, decentralized architecture, slicing data and distributing it across multiple nodes in the form of objects. It ensures load and capacity balance across each node, accesses and manages data through network connections, and provides a unified storage interface. Because the distributed storage system adopts a decentralized architecture, it is highly scalable. The failure of a single storage server node will not cause the entire distributed storage cluster to become unavailable. It supports dynamic expansion, adding, or removing storage server nodes, and can automatically relocate data objects in the cluster to balance storage space utilization across storage servers. The distributed storage client initiates an object I / O request to the object storage gateway. The gateway distributes the request to a unified, self-controlled, and scalable distributed storage consistency management system through a scalable hash calculation algorithm. The management system stores the object's index information, object metadata, and data in the storage pool. In the storage pool, the client's data is stored in slices, which are fixed to 4MB objects by default. The objects are grouped, and the group ID (unique identifier) ​​is the remainder of the Hash (object x) and the number of groups. Finally, the objects are stored in the underlying device NVMe SSD or HDD. The distributed storage system architecture includes multiple components, such as object gateway service, a unified, self-controlled, and scalable distributed storage consistency management system, storage pool, a controllable, scalable, distributed data balancing algorithm, and metadata and monitoring service clusters.

[0028] Specifically, the distributed storage system architecture includes the following key components: (1) Object Gateway Service: Provides a unified storage interface to provide access interface services for distributed storage clients.

[0029] (2) Unified, self-controlled, and scalable distributed storage consistency management system: This component is the core component of the distributed storage cluster. It provides distributed storage services in the form of objects. In this component, data is stored as objects. Each object has a unique identifier and related data. It is responsible for distributing the objects on each node of the storage cluster and providing data replication, recovery, load balancing and other functions to ensure data reliability and high-performance access.

[0030] (3) Storage pool: A distributed storage cluster consists of multiple storage nodes. Each storage node can contain multiple hard disks or storage devices. The storage pool divides the storage resources to form different pools to meet different storage needs. The data storage pool is composed of HDDs (hard disk drives) considering the cost. The metadata storage pool is composed of NVMe SSDs to ensure efficient metadata access operations. Data and metadata are stored in the form of objects (object storage devices, referred to as OSDs).

[0031] (4) Controllable, scalable, distributed data balancing placement algorithm: This component distributes data objects evenly across the nodes of the storage cluster to avoid data hotspots and improve system performance.

[0032] (5) Monitoring service cluster: The monitoring service cluster is responsible for monitoring the status and configuration information of the cluster. Its technical principles include maintaining the cluster's status information, configuration information, health status, etc. to ensure the consistency and availability of the cluster.

[0033] The improvements of this application are mainly applied to the OSD (Object Storage Device, which stores data in the form of objects on the physical disk of each node in the cluster) module in the distributed storage system. OSD is the core component responsible for storing data in the distributed storage system. It directly processes the client's IO requests. In the OSD module, the IO scheduling algorithm has the above-mentioned shortcomings. This application improves the IO scheduling algorithm in the OSD module so that it can dynamically adjust resource allocation according to the client's real-time IO usage and sudden demand, thereby improving the system's performance and resource utilization.

[0034] like Figure 2 As shown, the embodiment of the present application provides a resource adjustment method, which is applied to Figure 1 The distributed storage system in the example is used to illustrate the process, which includes the following steps: S1: Initialize the resource allocation parameters of the target client.

[0035] It should be noted that the target client refers to one of the multiple clients corresponding to the distributed storage system, and the resource allocation parameters include the reserved bandwidth r c , burst bandwidth b c , Total IO quota IOQ c and priority P c , and initialize the total available bandwidth C, time series period T, burst weight adjustment coefficient α, and maximum burst duration T burst and the minimum bandwidth threshold B min And the cumulative IO usage of each client IOU c Initialized to 0, where initialization means assigning variables to default values, that is, assigning values ​​to the above parameters. The unit of time series period T is seconds, which is a time unit that measures the resource usage of the client over a period of time, such as 300 seconds, 600 seconds, etc. IOQ c represents the total IO quota of client c in a time series period T, reflecting the maximum IO amount allowed to be used by the client in the period, r c The reserved bandwidth for client c is measured in IOPS (input and output operations per second). This is the basic bandwidth that the client can always obtain. c is the burst bandwidth of client c, also in IOPS, used to meet the client's burst IO demand at a specific moment, IOU c Refers to the cumulative IO usage of client c up to time t, recording the IO usage of the client from the beginning to the current moment; ω c (t) is the dynamic weight of client c at time t, which changes in real time according to the client's IO usage. C is the total available bandwidth of the system in IOPS, which represents the maximum IO bandwidth that the system can provide at a certain moment. c Indicates the priority of client c. The value range is 1-5. The larger the number, the higher the priority. It can be pre-set based on factors such as the client's business importance.

[0036] S2: Obtain state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0037] It should be noted that the state parameter refers to the request rate of the client at different time nodes. The required bandwidth corresponding to the request can be determined by the request rate and the corresponding request size; the burst state refers to the occurrence of sudden IO requests, that is, sudden high traffic.

[0038] S3: In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client.

[0039] It should be noted that resource adjustment parameters generally include dynamic weights and allocated bandwidth, and the allocated bandwidth is adjusted to meet the request processing requirements of the target client.

[0040] S4: adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters.

[0041] It should be noted that the allocated resources generally refer to the idle resources of the OSD, which are allocated to the target client to process the input and output requests of the target client.

[0042] In the above implementation, the weight can be dynamically adjusted according to the client's real-time IO resource usage, burst status and priority, which can better meet the differentiated IO needs of multiple clients. Whether it is a stable low-latency demand or a periodic burst demand, it can get a timely and effective response, and has strong dynamic adaptability; this application dynamically adjusts the resource allocation strategy to flexibly and reasonably allocate bandwidth according to the real-time needs of each client, so that the system can run more efficiently and provide services to more clients; in a distributed object storage environment, under the same hardware resource conditions, the optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response delay, and improves data reading and writing speed, thereby improving the performance of the entire distributed storage system.

[0043] In some specific implementations, initializing resource allocation parameters of the target client includes: Determine whether the current time series period is the initial time series period; In response to the current time series period being the initial time series period, defining the initial resource allocation parameter as the resource allocation parameter; In response to the current time series period not being the initial time series period, the resource allocation parameter of the previous time series period is adjusted based on the resource usage of the previous time series period, and the adjusted resource allocation parameter is defined as the resource allocation parameter.

[0044] Specifically, the initial time series cycle refers to the time series cycle in the first time series cycle when the system just starts running. The initial resource allocation parameter refers to the initial default value of the resource allocation parameter. When the current time series cycle is the first time series cycle, the initial default value of the resource allocation parameter is defined as the resource allocation parameter. When the current time series cycle is not the first time series cycle, the resource allocation parameter is adjusted according to the resource usage of the previous time series cycle, and the adjusted resource allocation parameter is defined as the resource allocation parameter of the current time series cycle. The initialization process is to determine the reserved bandwidth r of each client c according to the client's needs and system resource conditions. c , burst bandwidth b c , Total IO quota IOQ c and priority P c For example, for clients with higher latency requirements, a higher reserved bandwidth r can be allocated c and a higher priority P c For clients with sudden demands, increase the burst bandwidth b according to the demand c At the same time, determine the total available bandwidth C of the system, the time series period T, the burst weight adjustment coefficient α, and the maximum burst duration T burst and the minimum bandwidth threshold B min , where the total available bandwidth C of the system depends on the hardware configuration of the storage system, the time series period T can be set according to actual business needs; the burst weight adjustment coefficient α can be set to 0.5, and the maximum burst duration T burst Can be set to 30 seconds, minimum bandwidth threshold B min It can be set according to the actual situation of the system, and the cumulative IO usage of each client IOU c (0) Initialized to 0 to prepare for subsequent IO usage statistics.

[0045] In the above embodiment, the initial resource allocation parameters of the time series period are determined according to different time series periods, so that the resource allocation value can be determined more accurately, thereby improving resource utilization and dynamic adaptability of the system.

[0046] In some embodiments, such as Figure 3 As shown, before obtaining the status parameters of the target client, the resource adjustment method further includes: Detect whether the target client has issued an input / output request. If so, receive the input / output request issued by the target client. The input / output request refers to an IO request. The client initiates an IO request and sends the IO request to the MDS (Meta Data Server) of the distributed storage system. The MDS is responsible for managing the metadata of the file system. After receiving the request, the MDS determines the corresponding OSD set based on the file path and metadata information of the request and forwards the request to the corresponding OSD. After receiving the IO request from the MDS (one of the components of SQL Server, mainly used to manage the master data integration of enterprise data) through the OSD, it puts it into the global request queue. The global request queue is responsible for storing pending requests and ensures that each request is processed in order through the FIFO mechanism (First Input First Output, a first-in-first-out data buffering mechanism). When multiple requests arrive concurrently, the queue can effectively manage the order of these requests to avoid processing confusion. In response to receiving an input / output request from a target client, a state parameter of the target client is obtained, and a current time is recorded, wherein the state parameter includes a request rate and a request size of the IO request.

[0047] In the above embodiment, the state parameter of the target client is obtained by detecting whether an input / output request is received, so as to be used for subsequently determining the state of the target client.

[0048] In some specific implementations, determining whether the state of the target client is a sudden state based on the state parameter of the target client includes: Obtain the required bandwidth corresponding to the request rate of the input and output requests issued by the target client and the reserved bandwidth of the target client. The required bandwidth can be calculated based on the request rate and request size. The calculation method is a common method and the specific calculation process is not repeated here. In response to the required bandwidth being greater than the reserved bandwidth by a preset multiple, the target client is determined to be in a burst state, and a burst state flag is set for the target client, wherein the preset multiple can be set according to actual needs, such as 2 times, etc. The burst state flag is S c (t)=1, specifically, monitor the IO request rate of each client c in real time to determine whether the required bandwidth corresponding to the IO request rate of client c exceeds its reserved bandwidth r c If it exceeds 2 times, S c (t) is set to 1, and the start time t of the burst state is recorded start ; In response to the required bandwidth being less than or equal to the preset multiple of the reserved bandwidth, it is determined that the state of the target client is not a burst state, and a non-burst state flag is set for the target client, wherein the non-burst state flag is S c (t) = 0, where the IO request rate of client c corresponds to the required bandwidth that does not exceed its reserved bandwidth r c 2 times, then S c (t) is set to 0.

[0049] In some specific real-time manners, the resource adjustment method further includes: Obtain clients that have experienced mutation status within multiple time series periods and mark them based on time nodes. The number of time series periods can be set according to actual needs. In response to the number of occurrences of the same time node exceeding a preset threshold, the time node is marked once. By determining the duration of the mutation state of the time node in different time series periods of the client, the preset threshold can be set according to actual needs; In response to the number of time series periods corresponding to the duration of the mutation state being within a preset range being greater than a preset threshold, the time node is marked again. The preset range can be set according to actual needs, that is, the duration of the mutation state is greater than a preset value; When the time node of the current time series period reaches the time node marked twice, it is defined that the state of the target client at the time node is in a burst state.

[0050] In the above embodiment, by marking the time nodes where the burst state occurs multiple times, when the next time series period reaches the time node, the target client is directly defined as being in the burst state, thereby improving the efficiency of resource adjustment.

[0051] In some specific implementations, after determining that the state of the target client is a burst state and setting a burst state flag for the target client, the resource adjustment method further includes: Get the start time of the burst state, i.e. t start , and based on the start time, determine the duration of the burst state; In response to the duration of the burst state being greater than the preset burst state time, the state of the target client is defined as a non-burst state, and the state flag of the target client is reset to a non-burst state flag, wherein the preset burst state time is the maximum burst duration T start The specific value of the maximum burst duration can be set according to actual needs, that is, the duration of the burst state exceeds the maximum burst duration T start , then S c (t) is reset to 0.

[0052] In the above embodiment, the state of the target client is judged to determine whether it is a sudden state, so as to adjust the resources of the target client in the sudden state, so that the client can run stably, improve the dynamic adaptability of the system, and when the sudden state lasts for a long time, the flag is reset to 0 to accurately measure the client resource demand to avoid a client occupying too many resources for a long time, thereby improving resource utilization.

[0053] In some specific implementations, in response to the target client being in a burst state, determining the resource adjustment parameter of the target client includes: Determining a dynamic weight of the target client for determining bandwidth allocation based on resource usage of the target client; Determine the allocated bandwidth for the target client based on the dynamic weight.

[0054] In some specific implementations, determining the dynamic weight of the target client for determining bandwidth allocation based on the resource usage of the target client includes: Get the resource usage IOU of the target client from the start time to the current time in the time series period c (t), and the reserved bandwidth r of the target client c , where the time period from the start time to the current time is t; Based on resource usage IOU c (t) and the reserved bandwidth r of the target client c ,determine the dynamic weight of the target client to determine the allocated bandwidth.

[0055] In some specific embodiments, the target client is used to determine the dynamic weight of the allocated bandwidth, that is, the dynamic weight ω of the target client when time reaches t c (t), including: ; Among them, ω c (t) represents the dynamic weight, S c (t) represents the state identifier of the target client c, α represents the preset burst weight adjustment coefficient, P c represents the priority of the target client c, t represents the time, b c represents burst bandwidth, T represents time series period, IOQ c (t) represents the total input and output quota within the time series period, wherein the preset burst weight adjustment coefficient is used to increase the dynamic weight of the client when a burst IO situation occurs, so as to allocate more resources to it. For example, when a burst IO situation occurs on client c (S c (t)=1), its dynamic weight will increase accordingly, and combined with the client's priority Pc , and further adjust its priority in resource allocation.

[0056] In some specific implementations, determining the allocated bandwidth for the target client based on the dynamic weight includes: Based on the bandwidth calculation function and dynamic weight, the allocated bandwidth of the target client in the arrival time period t is determined. The bandwidth calculation function includes: ; Among them, B c (t) represents the allocated bandwidth, n represents the number of clients, ω j (t) represents the weight value of the jth client, and C represents the total available bandwidth value of the system.

[0057] For example, for each client c, ω is calculated according to the dynamic weight calculation formula c (t), for example, if the IOU of client c c (t) is less than or equal to r c *t and S c (t)=1,P c =3, then use Calculate the weight and calculate the bandwidth B allocated to each client according to the bandwidth allocation formula c (t), for example, there are 3 clients in the system, ω1(t)=0.6 for client 1, ω2(t)=0.3 for client 2, ω3(t)=0.1 for client 3, and the total available bandwidth of the system C=1000IOPS, then the bandwidth allocated to client 1 is , if the calculated Less than the minimum bandwidth threshold B min , then set B1(t) to B min , and so on, no more details.

[0058] In some specific implementations, adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client includes: In response to the allocated bandwidth of the target client being greater than or equal to a preset bandwidth threshold, defining the allocated bandwidth as a resource adjustment parameter of the target client to adjust the resource allocation parameter of the target client; In response to the allocated bandwidth of the target client being less than a preset bandwidth threshold, the preset bandwidth threshold is defined as a resource adjustment parameter of the target client, so as to adjust the resource allocation parameter of the target client.

[0059] The preset bandwidth threshold can be set according to actual needs and is generally the minimum bandwidth threshold.

[0060] In the above implementation, the dynamic weight of each client is determined by the weight calculation function, and the total available bandwidth of the allocation system is determined based on the dynamic weight to ensure the fairness and dynamism of resource allocation. At the same time, in order to avoid a client being unable to process requests normally due to the bandwidth allocated to the client being too small, a minimum bandwidth threshold B is set. min , if the calculated B c (t) is less than B min , then set the dynamic weight to B min , which avoids the impact of insufficient bandwidth allocated to the client on request processing and improves the overall performance of the system.

[0061] In some specific implementations, allocating resources to the target client according to the adjusted resource allocation parameters includes: Based on the allocated bandwidth, the target client's input and output requests are obtained from the global request queue and placed into the pending queue; In response to the existence of idle resources, the input and output requests of the target client are taken out from the waiting queue for processing, and the resource usage of the target client is updated.

[0062] Specifically, when an IO request arrives, the current time t is recorded, and a burst state detection is first performed to determine the S of each client. c (t) value, then, for each client c, calculate ω according to the above dynamic weight calculation formula c (t), and then calculate the bandwidth B allocated to each client according to the bandwidth allocation formula c (t), take each client's IO request from the global request queue according to the allocated bandwidth and put it into the corresponding waiting queue. When OSD has free resources, take the request from the waiting queue for processing. In the process of processing the request, update the IOU c , OSD is responsible for processing IO requests in real time and dynamically adjusting resource allocation according to the actual situation of the client. If the IO request rate of the client changes during the processing, resulting in a sudden change in the state, the OSD will be updated in time. c (t) and the value of C, and recalculate the dynamic weight and allocated bandwidth.

[0063] In the above implementation, dynamic weights and bandwidth allocation formulas are used in combination with client priorities to ensure that each client can obtain fair resource allocation at different stages, avoiding the problem of uneven resource allocation caused by unreasonable weight settings in related algorithms, ensuring the interests of each client, and for applications such as real-time monitoring systems and online games that are sensitive to latency, as well as big data analysis and file transfers that have high throughput requirements, resources can be reasonably allocated according to their needs, thereby improving the overall performance of the system.

[0064] In some specific implementations, the resource adjustment method further includes: In response to the current time series period having ended, obtaining resource usage of the target client in the current time series period; In response to the target client's resource usage being greater than the total input and output quota, the reserved bandwidth and the burst bandwidth for the next time series period are adjusted downward, and the target client's resource usage is reset; In response to the target client's resource usage being less than or equal to the total input and output quota, the reserved bandwidth and the burst bandwidth for the next time series period are increased and adjusted, and the target client's resource usage is reset.

[0065] Specifically, OSD stores data in the local disk according to the type of request (such as write request). During the storage process, it follows the data redundancy and consistency strategy of the distributed storage system to ensure data reliability. When the request processing is completed, OSD returns the processing result to MDS, and MDS returns the result to the client. After the client receives the response, the IO operation is completed. At the end of each time series period T, the IOU of each client is checked. c Whether the IOQ is exceeded c If it exceeds, the client's reserved bandwidth and burst bandwidth will be adjusted in the next billing cycle, that is, the client's reserved bandwidth will be reduced in the next billing cycle. and burst bandwidth For example, the reserved bandwidth is reduced by 10% and the burst bandwidth is reduced by 20% to punish clients that overuse resources. At the same time, the accumulated IO usage of each client is reset. c =0, a new time series cycle begins, thereby ensuring that the client uses resources reasonably within a time series cycle and maintaining the fairness and stability of the system. If it does not exceed, the reserved bandwidth and burst bandwidth of the client can be appropriately increased according to the situation, such as increasing the reserved bandwidth by 5% and the burst bandwidth by 10% to reward clients that use resources reasonably. At the same time, the cumulative IO usage of each client IOQ is reset c =0 and start a new time series period to ensure that the statistics of resource usage start again.

[0066] In the above implementation, by comparing the resource usage and total input and output quota of the previous time series period, the reserved bandwidth and burst bandwidth of the next time series period are adjusted, thereby improving the accuracy of resource allocation and further enhancing the overall performance of the system.

[0067] The resource adjustment method includes: initializing resource allocation parameters of a target client; obtaining status parameters of the target client, and determining whether the status of the target client is a burst state based on the status parameters of the target client; in response to the status of the target client being a burst state, determining resource adjustment parameters of the target client; adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters. The method has the following beneficial effects: (1) Strong dynamic adaptability: The algorithm can dynamically adjust the weight according to the client's real-time IO usage, burst status and priority. In actual application scenarios, when the client has a burst IO demand, such as database backup or video transcoding generating a large number of requests, the system can promptly perceive and increase its dynamic weight and allocate more resources to the client. Compared with the scheduling algorithm with fixed parameters, this application can better meet the differentiated IO requirements of multiple clients. Whether it is a stable low-latency demand or a periodic burst demand, it can get a timely and effective response.

[0068] (2) High resource utilization: By dynamically adjusting the resource allocation strategy, the system can make full use of the remaining bandwidth. The relevant algorithm easily leads to idle resources of some clients, while this application can allocate these idle resources to clients in need, reducing bandwidth waste. When multiple clients access concurrently, the bandwidth can be flexibly allocated according to the real-time needs of each client, enabling the system to run more efficiently and provide services to more clients.

[0069] (3) Fairness is guaranteed: By using dynamic weights and bandwidth allocation formulas, combined with client priorities, each client can obtain fair resource allocation at different stages, avoiding the problem of uneven resource allocation caused by unreasonable weight settings in related algorithms, and ensuring the interests of each client. For applications such as real-time monitoring systems and online games that are sensitive to delays, as well as big data analysis and file transfers that have high throughput requirements, resources can be reasonably allocated according to their needs.

[0070] (4) Improved system performance: In a distributed object storage environment, under the same hardware resource conditions, the overhead of distributed metadata occupying NVMe SSD space can be reduced. The optimized IO scheduling algorithm makes the system more efficient in processing client requests, reduces system response latency, and increases data reading and writing speeds, thereby improving the performance of the entire distributed storage system.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0072] It should be understood that although Figure 1-Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1-Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0073] In one embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a resource adjustment method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0074] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0075] An embodiment of the present application provides an electronic device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform steps in an embodiment of a resource adjustment method, including: S1: Initialize the resource allocation parameters of the target client.

[0076] S2: Obtain state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0077] S3: In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client.

[0078] S4: adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters.

[0079] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the steps in the resource adjustment method embodiment when running, including: S1: Initialize the resource allocation parameters of the target client.

[0080] S2: Obtain state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0081] S3: In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client.

[0082] S4: adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters.

[0083] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0084] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in the resource adjustment method embodiment are implemented, including: S1: Initialize the resource allocation parameters of the target client.

[0085] S2: Obtain state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0086] S3: In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client.

[0087] S4: adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters.

[0088] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps in the resource adjustment method embodiment are implemented, including: S1: Initialize the resource allocation parameters of the target client.

[0089] S2: Obtain state parameters of the target client, and determine whether the state of the target client is a sudden state based on the state parameters of the target client.

[0090] S3: In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client.

[0091] S4: adjusting the resource allocation parameters of the target client based on the resource adjustment parameters of the target client, and allocating resources to the target client according to the adjusted resource allocation parameters.

[0092] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0093] The above is a detailed introduction to a resource adjustment method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A resource adjustment method, characterized in that: The method comprises: Initialize resource allocation parameters of the target client; Acquire a state parameter of the target client, and determine whether the state of the target client is a sudden state based on the state parameter of the target client; In response to the state of the target client being a sudden state, determining a resource adjustment parameter of the target client; The resource allocation parameters of the target client are adjusted based on the resource adjustment parameters of the target client, and resources are allocated to the target client according to the adjusted resource allocation parameters.

2. The resource adjustment method according to claim 1, characterized in that: Initialize the resource allocation parameters for the target client including: Determine whether the current time series period is the initial time series period; In response to the current time series period being the initial time series period, defining an initial resource allocation parameter as the resource allocation parameter; In response to the current time series period not being the initial time series period, the resource allocation parameter of the previous time series period is adjusted based on the resource usage of the previous time series period, and the adjusted resource allocation parameter is defined as the resource allocation parameter.

3. The resource adjustment method according to claim 1, characterized in that: Before obtaining the status parameters of the target client, the method further includes: detecting whether the target client issues an input / output request, and if so, receiving the input / output request issued by the target client; In response to receiving the input / output request sent by the target client, the state parameters of the target client are acquired, and the current time is recorded.

4. The resource adjustment method according to claim 1, characterized in that: Determining, based on the state parameter of the target client, whether the state of the target client is an emergency state includes: Obtaining a required bandwidth corresponding to a request rate of an input / output request issued by the target client and a reserved bandwidth of the target client; In response to the required bandwidth being greater than the reserved bandwidth by a preset multiple, determining that the state of the target client is a burst state, and setting a burst state flag for the target client; In response to the required bandwidth being less than or equal to a preset multiple of the reserved bandwidth, it is determined that the state of the target client is not a burst state, and a non-burst state flag is set for the target client.

5. The resource adjustment method according to claim 4, characterized in that: After determining that the state of the target client is a sudden state and setting a sudden state flag for the target client, the method further includes: Obtaining a start time of the sudden state, and determining a duration of the sudden state based on the start time; In response to the duration of the burst state being greater than a preset burst state time, the state of the target client is defined as a non-burst state, and the state identifier of the target client is reset to a non-burst state identifier.

6. The resource adjustment method according to claim 1, characterized in that: In response to the state of the target client being a sudden state, determining the resource adjustment parameter of the target client includes: Determining a dynamic weight of the target client for determining bandwidth allocation based on resource usage of the target client; The allocated bandwidth of the target client is determined according to the dynamic weight.

7. The resource adjustment method according to claim 6, characterized in that: Determining, based on the resource usage of the target client, a dynamic weight of the target client for determining bandwidth allocation includes: Get the resource usage IOU of the target client from the start time to the current time in the time series period c (t), and the reserved bandwidth r of the target client c ; Based on the resource usage IOU c (t) and the reserved bandwidth r of the target client c , determining the dynamic weight of the target client for determining the allocated bandwidth.

8. The resource adjustment method according to claim 7, characterized in that: Determining the dynamic weight of the target client for determining bandwidth allocation includes: ； Among them, ω c (t) represents the dynamic weight, S c (t) represents the state identifier of the target client c, α represents the preset burst weight adjustment coefficient, P c represents the priority of the target client c, t represents the time, b c represents burst bandwidth, T represents time series period, IOQ c (t) represents the total input and output quota within the time series period.

9. The resource adjustment method according to claim 8, characterized in that: Determining the allocated bandwidth of the target client according to the dynamic weight includes: Determine the allocated bandwidth of the target client based on a bandwidth calculation function and the dynamic weight, wherein the bandwidth calculation function includes: ； Among them, B c (t) represents the allocated bandwidth, n represents the number of clients, ω j (t) represents the weight value of the jth client, and C represents the total available bandwidth value of the system.

10. The resource adjustment method according to claim 6, characterized in that: Adjusting the resource allocation parameter of the target client based on the resource adjustment parameter of the target client includes: In response to the allocated bandwidth of the target client being greater than or equal to a preset bandwidth threshold, defining the allocated bandwidth as a resource adjustment parameter of the target client to adjust the resource allocation parameter of the target client; In response to the allocated bandwidth of the target client being less than a preset bandwidth threshold, the preset bandwidth threshold is defined as a resource adjustment parameter of the target client, so as to adjust the resource allocation parameter of the target client.

11. The resource adjustment method according to claim 10, characterized in that: Allocating resources to the target client according to the adjusted resource allocation parameters includes: Based on the allocated bandwidth, obtaining the input and output requests of the target client from a global request queue, and placing the input and output requests of the target client into a queue to be processed; In response to the existence of idle resources, the input and output requests of the target client are taken out from the to-be-processed queue for processing, and the resource usage of the target client is updated.

12. The resource adjustment method according to claim 1, characterized in that: The method further comprises: In response to the current time series period ending, obtaining resource usage of the target client in the current time series period; In response to the target client's resource usage being greater than the total input and output quota, reducing and adjusting the reserved bandwidth and burst bandwidth for the next time series period, and resetting the target client's resource usage; In response to the target client's resource usage being less than or equal to the total input and output quota, the reserved bandwidth and the burst bandwidth for the next time series period are increased and adjusted, and the target client's resource usage is reset.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the resource adjustment method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the resource adjustment method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the resource adjustment method according to any one of claims 1 to 12 are implemented.

Images