Resource management and control method, electronic device, storage medium and program product

Abstract

The present invention provides a resource control method, electronic device, storage medium and program product, wherein the method includes: receiving an access request of a distributed file system; determining a target user in a client to which the access request belongs, and obtaining control information for resource control of the target user from a metadata mapping MAP; determining a scheduling strategy for the target user based on the control information of the target user, and performing service quality QoS control on the access request based on the scheduling strategy. By determining the target user in the client to which the access request belongs, and obtaining the control information of the user corresponding to the access request stored in advance from the MAP, the acquisition process of the control information of the access request is realized, and only the relationship between the access request and the user is concerned. Therefore, the method can constrain any request issued by the client user interacting with it, realize the user-granular control process of all requests, and improve the control granularity of resources.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a resource management and control method, electronic equipment, storage medium and program product. Background Art

[0002] The Quality of Service (QoS) strategy of the Ceph Filesystem (CephFS) mainly ensures the stability and efficiency of storage performance through a series of parameter configurations and mechanisms.

[0003] In the related art, when implementing the QoS policy of CephFS, it is implemented based on the control and constraint of the logical volume SubVolume. Since SubVolume is an independent storage unit, the QoS control method directly relying on the storage unit is relatively coarse-grained, resulting in low control granularity. Summary of the invention

[0004] The present invention provides a resource management method, an electronic device, a storage medium and a program product, which are used to solve the defect that the QoS management method directly based on the management and constraint of the logical volume SubVolume in the related technology is relatively coarse-grained, resulting in low management granularity, and realizes user-granular QoS management of all requests, thereby improving the management granularity of resources.

[0005] The present invention provides a resource management method, comprising the following steps: Receive access requests to the distributed file system; Determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Based on the control information of the target user, a scheduling policy of the target user is determined, and based on the scheduling policy, quality of service QoS control is performed on the access request.

[0006] According to a resource control method provided by the present invention, based on the scheduling strategy, the access request is subjected to quality of service (QoS) control, including: The access request is added to the scheduling queue, and the scheduling policy is registered in the scheduling queue, so that after the consumer thread of the distributed file system obtains the access request from the scheduling queue, based on the registered scheduling policy, the pre-registered request processing function associated with the access request is executed according to the mClock algorithm to perform QoS control of the access request.

[0007] According to a resource management method provided by the present invention, after receiving the access request of the distributed file system, the method further includes: In the case where the access request is an initial access request of the target user, determining user information of the target user based on parsing metadata of a client to which the access request belongs; The user information and the management and control information corresponding to the user information are added to the MAP.

[0008] According to a resource management method provided by the present invention, before acquiring the management information for resource management of the target user from the metadata map MAP, the method further includes: It is determined that the access request is a request to enable the mClock algorithm to execute the QoS policy.

[0009] According to a resource management method provided by the present invention, after receiving the access request of the distributed file system, the method further includes: In the case where it is determined that the access request is not a request for enabling the mClock algorithm to execute the QoS policy, a processing function corresponding to the access request is called to execute a processing procedure of the access request.

[0010] According to a resource management method provided by the present invention, before receiving an access request to a distributed file system, the method further includes: It is determined that a connection has been established with the client to which the access request belongs, and that authentication has been passed and a legal network session identifier has been obtained.

[0011] According to a resource management method provided by the present invention, the management information at least includes a minimum service number, a maximum service number and a service priority ratio.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the resource management methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the resource management and control method described in any one of the above is implemented.

[0014] The present invention also provides a computer program product, including a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the resource management and control methods described above.

[0015] The resource control method, electronic device, storage medium and program product provided by the present invention realize the process of obtaining the control information of the access request by determining the target user in the client to which the access request belongs and obtaining the control information of the user corresponding to the access request pre-stored in the MAP. Based on the obtained control information, a scheduling strategy for the access request is generated to realize the scheduling control process of the access request, and only pays attention to the relationship between the access request and the user. Therefore, the method can constrain any request issued by the client user interacting with it, realizes the user-granular control process of all requests, and improves the control granularity of resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of the resource management and control method provided by the present invention.

[0018] Figure 2 It is a schematic diagram of the interaction flow between users of multiple clients and a server provided by the present invention.

[0019] Figure 3 It is a schematic diagram of simulation results provided by the present invention.

[0020] Figure 4 It is a structural schematic diagram of the resource management and control device provided by the present invention.

[0021] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] CephFS is the file system layer of the Ceph distributed storage system, which provides a POSIX-compliant distributed file system that is highly available, scalable, and has strong fault tolerance.

[0024] QoS policy is a set of technologies and mechanisms used in computer networks to manage and optimize network resources to provide a specific level of service quality. QoS policy aims to ensure that the network can provide the required quality of service when it is fully loaded or congested by performing priority management and resource allocation for different types of data traffic. It allows network administrators to distinguish traffic from different applications and services and provide bandwidth guarantees for critical business traffic, thereby improving the overall performance of the network and user experience.

[0025] There are two main ways to implement CephFS QoS policies in related technologies: One of the related technologies is to implement the constraint process from the client side based on the token bucket, which constrains the number of read / write operations per second (IOPS) of a single client. It should be noted that IOPS is an important indicator for measuring the performance of storage devices, which reflects the ability of storage devices to deal with random read and write operations. By constraining and controlling IOPS, it is possible to reasonably allocate and optimize storage resources, thereby improving the quality of service QoS, and then implementing the QoS policy in CephFS.

[0026] The second related technology is to constrain the IOPS of the SubVolume of CephFS from the Metadata Server (MDS) side based on the mClock algorithm. Among them, SubVolume is a logical volume divided in the storage system, which is mainly used to divide and manage the storage space, making the storage and access of data more flexible and efficient. The mClock algorithm is an input / output (I / O) scheduling algorithm based on time tags. It is mainly used in the Ceph distributed storage system to achieve effective control of the IOPS or bandwidth of storage access, thereby achieving fine control of storage resources and ensuring that each client interacting with the distributed storage system can allocate resources according to the preset configuration. MDS is a key component in the CephFS file system. Its main responsibility is to manage the metadata of the CephFS file system. Metadata is data about data, which describes the structure, attributes and relationships between files and directories in the file system. MDS enables CephFS to provide POSIX-compatible file system services by maintaining these metadata.

[0027] One of the related technologies is to constrain the access IOPS of a single client based on Token Bucket from the client. The disadvantages of this solution include: Resource control lacks integrity: Since IOPS resource control is performed on the client side, it does not rely on and cannot perceive the pressure on the server side. As the number of clients increases, the IOPS limit parameters of each client need to be manually adjusted. Otherwise, the pressure on the server side may reach a bottleneck, eventually leading to abnormal resource control. The cost of transforming existing clusters is high: Since the control policy is located on the client, the client needs to be transformed and upgraded. Since the client may mount CephFS in kernel mode or in user mode, both mounting methods need to be upgraded. The cost of upgrading the kernel mode mounting method is particularly high, which requires recompiling the operating system kernel, which has a huge impact on online services.

[0028] The second related technology is to constrain the IOPS of the SubVolume of CephFS from the MDS side based on the mClock algorithm.

[0029] The second disadvantage of the related technology is: The scope of application scenarios is relatively small: Since it strictly relies on SubVolume for implementation, the business must use SubVolume to distinguish the mount directory items. However, most business usage scenarios may be based on K8S static mounting and mapping to the host file system path Host Path. In this method, the SubVolume feature of CephFS is not used, so this solution cannot be used in this scenario; For common directories that are not configured to be controlled by QoS policies, the QoS effect will be interfered with: Since the storage paths of SubVolumes are independent of each other, in order to access public data, the client needs to mount the public data directory or directly mount the top-level root directory. This will result in the client's access to specific directories not being subject to IOPS control, which will affect the access pressure on the server side and eventually cause the QoS policy to be seriously interfered with and fail; Low control granularity: The control is implemented based on the control constraints of the logical volume SubVolume. Since SubVolume is an independent storage unit, the QoS control method directly relying on the storage unit is relatively coarse-grained, resulting in a low control granularity for implementing the QoS policy.

[0030] In view of the defects in the related art, the present invention provides a resource management method. Figure 1 It is a flow chart of the resource management method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 110, receiving an access request of a distributed file system; Step 120, determining a target user in the client to which the access request belongs, and obtaining control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Step 130: Determine a scheduling policy for the target user based on the control information of the target user, and perform quality of service QoS control on the access request based on the scheduling policy.

[0031] The following takes the MDS in a distributed file system executing the resource management and control method provided by the present invention as an example to describe the technical solution of the present invention in detail.

[0032] In step 110, the MDS receives an access request of the distributed file system, wherein the received access request is sent by a client interacting with the distributed file system.

[0033] It should be noted that before the MDS receives an access request to the distributed file system, the client needs to establish a connection with the distributed file system.

[0034] Specifically, it can establish contact with the client based on the MON component in the distributed file system and obtain a legal network session identifier SessionID through authentication. The MON component will accept the client's request operation and provide a basis for subsequent operations. Among them, the MON component is a key component in the Ceph distributed file system, responsible for maintaining the status and metadata of the distributed file system.

[0035] In step 120, after receiving the access request to the distributed file system, the MDS parses the received access request to determine the client sending the access request and the authentication user used by the client. The target user is the authentication user used by the client. After determining the target user, the MDS queries the relevant information of the target user in the MAP to obtain the control information for resource control of the target user.

[0036] The metadata mapping MAP refers to a mapping relationship information, which includes user information about all users in the distributed file system. After the user establishes a connection with the distributed file system, the user information of the user will be added to the MAP.

[0037] The user control information in MAP is pre-added. It is understandable that MAP contains user information of all users interacting with the distributed file system. Before QoS policy control is performed on access requests, the control information of all users interacting with the distributed file system is added to MAP to achieve subsequent control of access requests of all users. The control information of pre-added users can be shown in Table 1: Table 1 Control information comparison table

[0038] Among them, R is the minimum service number, L is the maximum service number, and W is the service priority ratio.

[0039] It should be noted that the management and control information is constructed for users and is used to implement relevant information for QoS management and control of users. It can include the minimum service times, the maximum service times, and the service priority ratio. Among them, the minimum service times (Reservation), the maximum service times (Limit), and the service priority ratio (Weight) are key management and control factors. The minimum service times ensure that users get the most basic services and prevent request starvation. The maximum service times prevents excessive resource consumption and prevents a user or task from occupying too many resources. The service priority ratio schedules resources according to the importance and priority of the task, ensuring that critical tasks are given priority while preventing low-priority tasks from being completely ignored.

[0040] After receiving the access request, the MDS may determine the user information of the target user corresponding to the access request based on parsing the access request, and search in the MAP based on the user information to determine the management and control information corresponding to the user information.

[0041] In step 130, a scheduling policy of the target user is determined based on the control information of the target user, and quality of service QoS control is performed on the access request based on the scheduling policy.

[0042] MDS formulates scheduling strategies for target users based on the user's control information. The control information is the parameter information related to QoS control of the target user's access request. The scheduling strategy for the target user can be determined based on the setting of the numerical range corresponding to the relevant parameters of the control information. For example, when the control information is determined to be the minimum service number, the maximum service number, and the service priority ratio, the scheduling strategy for the target user can be determined based on the upper and lower limits of each parameter. The main purpose of the scheduling strategy is to reasonably allocate system resources, ensure user service quality, and prevent resource abuse or a task from occupying too many system resources.

[0043] Based on the scheduling strategy, MDS will perform QoS control on the target user's access request. The goal of QoS control can be to ensure that the access request can get the appropriate service according to the set priority, quota, delay requirements, etc.

[0044] Optionally, QoS control measures may include: Bandwidth Throttling: MDS can limit the bandwidth usage of each user through bandwidth control to ensure that the system does not occupy too much bandwidth due to requests from a single user under high load conditions.

[0045] Latency Control: In real-time applications, latency may be a very important QoS indicator. MDS prioritizes low-latency tasks based on user service requirements. For low-priority user requests, MDS may introduce a queuing mechanism to delay the processing of their requests to ensure low-latency responses for high-priority tasks.

[0046] Resource Isolation: To prevent different users’ tasks from interfering with each other, MDS may adopt a resource isolation strategy. By allocating an independent resource pool to each user or task, MDS can ensure that different users do not interfere with each other. Resource isolation measures may include: Request Queuing: When system resources are insufficient, MDS may put requests into a queue and sort them by priority or quota.

[0047] Rate Limiting: If a user exceeds their I / O quota or bandwidth limit, MDS may implement rate limiting or speed limiting to ensure that the user does not exceed the resource limits that the system can bear.

[0048] Based on the control information of the target user, MDS can ensure fair resource allocation and efficient operation of the system by formulating appropriate scheduling strategies. Through QoS control, MDS can dynamically adjust resource allocation according to user priority, quota, latency requirements, etc., ensuring user experience while avoiding excessive consumption or unfair use of system resources.

[0049] In the process of multiple clients interacting with the server to which the distributed file system belongs, the specific interaction process can be as follows: Figure 2 The schematic diagram of the interaction flow between multiple client users and the server provided by the present invention is shown. The clients specifically include client 1, client 2, client 3 and client 4. The users in the clients include user A, user B and user C. Among them, the users in the clients refer to the users authenticated by the distributed file system, and any user can be used when establishing a connection with the distributed file system, that is, user A, user B and user C can all be used to establish an authenticated connection with the distributed file system.

[0050] The interaction process between the authenticated user of the client and the server is as follows: the authenticated user logged in in the client can interact with the server to which the distributed file system belongs. After the authenticated user in the client sends a request to the server, the interactive operations corresponding to the request by the server include: receiving the request, parsing metadata, registering the request, and consuming the request. After the server completes the execution, it returns the response result to the corresponding client.

[0051] Based on the control information in Table 1, determine the control policy for user A, user B, and user C. After the control policy is enabled on the MDS side, control is performed on the above users A, user B, and user C. The control results can be as follows: Figure 3 The simulation result provided by the present invention is shown in the schematic diagram.

[0052] Figure 3 The horizontal axis represents the total IOPS that the distributed file system can carry, and the vertical axis represents the amount of IOPS that each user can allocate (use) under the current IOPS that the distributed file system can carry. The comparison of these two indicators is used to show how to allocate the corresponding IOPS to each client under the different total IOPS carried by the distributed file system. The specific allocation process can be: When the total IOPS of the distributed file system is 500: Since it can just meet the R configuration requirements of three users, in this scenario, the IOPS that can be allocated to each user are 200, 200, and 100, and the total is 500; When the total IOPS of the distributed file system is 1000, if the IOPS is allocated according to the weight, the IOPS that each user can obtain is: 166, 333, 500. In this case, the R requirement of 200 of user A cannot be met. Therefore, the IOPS cannot be simply allocated according to the W weight. The R requirement must be met first, and the remaining part is allocated according to the W weight. Therefore, 500 of 1000 must be allocated to three users first, 200, 200, and 100; then the remaining 500 is allocated according to the W weight ratio of 1:2:3, that is, the IOPS that can be allocated to each user is: 200+(1 / 6×500), 200+(2 / 6×500), 100+(3 / 6×500), that is, 283, 366, and 350. When the total IOPS of the distributed file system is 1500, if the IOPS is allocated according to the weight, each user can obtain 250, 500, and 750 IOPS. It is found that these values ​​are greater than the R value of each user and do not exceed the corresponding L value. Therefore, the final allocated IOPS is: 250, 500, and 750. When the total IOPS of the distributed file system is 2000, if the IOPS is allocated according to the weight, the IOPS that each user can obtain is: 333, 666, 1000. It is found that these values ​​are greater than the R value of each user and do not exceed the corresponding L value. Therefore, the final allocated IOPS is: 333, 666, 1000; When the total IOPS of the distributed file system is 2500, if the IOPS is allocated according to the weight, the IOPS that each user can obtain is: 416, 833, 1250; in this case, it is found that the value of user C, 1250, has exceeded its L limit, so we should limit the value of user C to 1000, and then allocate 2500 - 1000 = 1500 of the total IOPS, and then allocate it according to the W weight ratio of user A and user B, that is, each user is allocated 1500×(1 / 3), 1500×(2 / 3), that is, 500, 1000, and finally the IOPS that can be allocated to each user is: 500, 1000, 1000; When the total IOPS carried by the distributed file system is 3000, in the same scenario as 2500, we should limit the L value of user C, so user C is limited to 1000, and the remaining 3000-1000=2000 will be distributed according to the W weight ratio of user A and user C, that is, each user gets 2000×(1 / 3), 2000×(2 / 3), that is, 666, 1333. Finally, the IOPS allocated to each user is: 666, 1333, 1000.

[0053] Since mClock controls the execution of each corresponding request at multiple intervals, for example, for user A, the minimum number of requests R is limited to 200. For each request initiated by user A, mClock will execute one every 1 / 200 seconds. In this way, it will be found that it takes a total of 1 second to execute 200 requests. Therefore, based on the control process of each access request, the overall IOPS control of the client and the corresponding IOPS control of each user are indirectly realized. Since only the relationship between the access request and the user is concerned, any request issued by the client user interacting with the distributed file system can be constrained, realizing the user-granular control process of all requests.

[0054] It can be understood that based on the QoS control process of access requests implemented from the MDS side, the following can be achieved: Solve the problem of lack of integrity in resource control: Since IOPS resources are allocated and controlled from the MDS side of the distributed file system, the problem of over-independence of client-side control is solved, and global resource allocation can be performed based on the overall performance of the server, providing overall resource control granularity; Solve the problem of high transformation cost of existing clusters: Since the IOPS resource is allocated and controlled from the MDS side of the distributed file system, there is no need to transform and upgrade the client. This is very friendly to existing clusters, especially historical existing clusters with a large number of clients and a large number of users. In order to use this feature, you only need to continue to upgrade the server, and the overall change has little impact. Solve the problem of too small application scope of application scenarios: Since each client needs to apply for a user ID when mounting CephFS resources, it is perfectly compatible with the use of SubVolume scenarios, kernel state or user state mounting methods, and host or container access, and has a wide range of application scenarios and scopes; Solve the problem that ordinary directories that are not configured to be controlled by QoS policies will interfere with the QoS effect: Since it is a QoS policy based on user granularity, the server side only focuses on the relationship between the request and the user when executing the QoS scheduling policy, and no longer distinguishes the directory requested to access. Therefore, this method can constrain any request issued by all clients, thereby achieving control over all requests and solving the problem of interference of non-controlled directories on controlled directories in the SubVolume solution.

[0055] The resource control method provided by the present invention realizes the process of obtaining control information of the access request by determining the target user in the client to which the access request belongs and obtaining the control information of the user corresponding to the access request pre-stored from the MAP during the process of executing the QoS control of the distributed file system. Based on the obtained control information, a scheduling strategy for the access request is generated to realize the scheduling control process of the access request, and only pays attention to the relationship between the access request and the user. Therefore, the method can constrain any request issued by the client user interacting with it, realizes the user-granular control process of all requests, and improves the control granularity of resources.

[0056] Based on the above embodiment, step 130 specifically includes: based on the scheduling policy, performing service quality QoS control on the access request, including: adding the access request to the scheduling queue, and registering the scheduling policy in the scheduling queue, so that after the consumer thread of the distributed file system obtains the access request from the scheduling queue, based on the registered scheduling policy, executes the pre-registration request processing function associated with the access request according to the mClock algorithm to perform the QoS control process of the access request.

[0057] Before scheduling and controlling, you need to make sure that the consumer thread of mClock QoS has been started and each request in the consumer thread has associated control information.

[0058] Specifically, MDS adds the received access request to a scheduling queue and registers the scheduling policy in the scheduling queue. This scheduling queue acts as a cache, temporarily storing all pending access requests and processing requests in order and priority. Among them, the scheduling queue is a task queue for processing access requests. Registering a scheduling policy in the scheduling queue is to add an access request task to the task queue, and the request task will carry the control information of the corresponding access request for the subsequent use of the mClock algorithm.

[0059] In a distributed file system, a consumer thread is a thread that takes a request from a scheduling queue and executes it. After extracting a request from a scheduling queue, the consumer thread will decide how to process the request based on the scheduling policy registered for the request. The goal of the consumer thread is to execute the pre-registered request processing function associated with the request based on the control information of the access request, so as to achieve resource allocation and service quality assurance. Among them, the pre-registered request processing function is the processing function corresponding to the MDS processing these access requests.

[0060] The consumer thread gets requests from the scheduling queue in sequence. The thread will take out the requests according to the order or priority of the scheduling queue. Each request is associated with a request processing function (for example, I / O request processing function, data read / write function, etc.) when it is registered. The consumer thread will execute these processing functions according to the scheduling policy. The scheduling policy determines how to process the access request.

[0061] When the consumer thread executes a request, it processes the request in the queue according to the mClock algorithm. QoS control is designed to ensure that the system provides resources that meet the expected service quality according to the scheduling strategy.

[0062] After the QoS control process of the access request is completed, the consumer thread will return the result of the request processing.

[0063] By adding access requests to the scheduling queue and registering relevant scheduling policies, MDS can ensure that the system processes each request according to the predetermined service quality standards. The consumer thread executes the corresponding request processing function according to the request priority and scheduling policy in the scheduling queue to complete QoS management tasks such as bandwidth control, delay management, and resource allocation. The mClock algorithm and dynamic adjustment mechanism ensure the efficient use and fair allocation of system resources, and guarantee the service quality of access requests.

[0064] Based on the above embodiment, step 110 specifically includes: after receiving an access request for a distributed file system, it also includes: when the access request is the first access request of the target user, determining the user information of the target user based on parsing the metadata of the client to which the access request belongs; adding the user information and the management and control information corresponding to the user information to the MAP.

[0065] The initial access request refers to the first request of the user in the current session or system life cycle. When MDS receives the initial access request, it first needs to parse the client metadata in the request to obtain user information.

[0066] Client metadata usually includes the following: Client ID, which identifies the client device or application that initiated the request. User ID, which uniquely identifies the target user accessing the file system. Request timestamp, which is the time when the request was initiated and may be used to queue and schedule with other access requests. Request type and parameters: such as read and write requests, which may affect the resource scheduling strategy.

[0067] MDS determines user information based on the parsed client metadata and user ID. It should be noted that each client will use a user for authentication when connecting to the distributed file system, and this authentication information will be stored in MDS, so MDS can parse the corresponding user information based on the access request. After obtaining and parsing the user information, MDS can store the user information and control information in the form of key-value pairs in the metadata map MAP.

[0068] Optionally, MDS not only stores this information, but also regularly maintains and updates the user's status. The user's status may change with usage, number of requests, resource quota, priority, etc. Therefore, MDS needs to monitor the user's access and dynamically adjust the control information according to actual needs.

[0069] It is understandable that for subsequent access requests from the user, the MDS no longer needs to reparse the client metadata to determine the user information, but directly reads the user's control information from the MAP. Based on this information, the MDS will decide how to schedule requests, allocate resources, and perform related QoS control.

[0070] Based on the above embodiment, step 120 specifically includes: before obtaining the control information for performing resource control on the target user from the metadata mapping MAP, it also includes: determining that the access request is a request for enabling the mClock algorithm to execute the QoS policy.

[0071] Before obtaining the control information for resource control of the target user from the metadata map MAP, it is necessary to ensure that the mClock QoS policy has been enabled, the MDS is in the active state, and the control information associated with the corresponding access request can be found.

[0072] Among them, whether to enable the QoS scheduling policy is a global configuration of MDS. If the MDS enables the QoS scheduling policy, all client requests received by the MDS will be subject to QoS scheduling. Directly obtaining the global configuration of the MDS can determine whether the MDS has enabled the QoS scheduling policy. The status information of the MDS is also a global monitoring indicator of the MDS. Just like whether the QoS scheduling policy of the MDS is enabled as a global configuration, it can be obtained through the member variables inside some structures of the MDS.

[0073] In the case where it is determined that the access request is not a request for enabling the mClock algorithm to execute the QoS policy, a processing function corresponding to the access request is called to execute a processing procedure of the access request.

[0074] Specifically, the MDS selects different scheduling mechanisms according to whether the mClock QoS policy is enabled.

[0075] If the mClock QoS policy is not enabled, the MDS will use the default processing function to handle the client's access request. In this case, it will be executed according to the standard, traditional request processing process.

[0076] The specific execution process is as follows: the client initiates an access request. After receiving the request, the MDS needs to parse the content of the request, such as the resource type requested, the type of operation requested (read, write, delete, etc.), and the target resource requested. MDS calls the corresponding default processing function according to the type of request. The default processing function usually involves standard resource access control, permission verification, resource allocation, and other tasks. The default processing function will directly perform the corresponding operation based on the content of the request. The processing process does not perform complex scheduling of the priority and latency of the request, but directly processes the requests in the order in which they arrive. After the default processing function is executed, the MDS will return the corresponding results (such as file reading results, confirmation of write operations, etc.) to the client. The entire process usually does not involve too much queuing or scheduling.

[0077] When the mClock QoS policy is enabled, the MDS will enter a more complex scheduling mechanism. At this time, all client access requests will be queued and prioritized according to the mClock algorithm to ensure that the system can meet the quality of service (QoS) requirements of different clients and optimize resource allocation and use.

[0078] After the client initiates a request, the MDS first captures and parses the request. The MDS arranges the processing order of the request according to the rules of the mClock algorithm. The request waits for execution in the scheduling queue. When it is the turn of a request, the system calls the corresponding scheduler to execute the request. At this time, the scheduler not only needs to consider the type of request, but also needs to refer to the defined scheduling policy. Once the scheduler determines the execution order of a request, it calls the corresponding pre-registered request processing function to execute the request. Unlike the default processing function, the execution process of these functions will be based on the scheduling policy to achieve service quality control. After the request is processed, the MDS returns the result to the client. During this process, the system will try to ensure that the service quality requirements are met, such as the request is completed within the specified time and the throughput reaches the expected value.

[0079] Based on the above embodiment, step 110 specifically includes: before receiving the access request of the distributed file system, it also includes: determining that a connection has been established with the client to which the access request belongs, and passing the authentication and obtaining a legal network session identifier.

[0080] For a distributed file system, the communication between the client and the server to which the distributed file system belongs must be verified to ensure that both parties are trustworthy. As a distributed resource management system, a distributed system usually maintains an effective connection between the client and the server through a session management mechanism.

[0081] When a client accesses a distributed file system, it first needs to initiate a connection request to the system's entry point through a network protocol. The connection request can carry some identification information for initializing the session.

[0082] The server will check the request initiated by the client to confirm that the request comes from a valid and known client. If it is the first request, the system will assign a new network session ID to the client. If it is a subsequent request, the client will carry the existing session ID to continue communicating with the system.

[0083] After the connection is established, the server to which the distributed file system belongs will perform further verification to ensure that the client's request is legitimate and authorized, which can be achieved through identity authentication and permission checking.

[0084] Before receiving access requests to the distributed file system, you must first ensure that a valid network connection has been established with the client and verify the client's identity through the authentication mechanism. After authentication, the system will assign a valid network session identifier to the client, which is used to identify the client in the system and track the client's access rights and requests. The session identifier is not only used for verification and resource allocation of subsequent requests, but also plays a role in session management and security protection, ensuring the security, availability, and reliability of the system.

[0085] For the above embodiment, it should be noted that the management and control information at least includes the minimum service times, the maximum service times and the service priority ratio.

[0086] The minimum service times refers to the minimum number of times each user, application or request can get service within a certain time period (such as every minute, hour, day, etc.). This policy is usually used to guarantee the basic service quality of certain users or requests, ensuring that they are not completely ignored or queued for too long.

[0087] The maximum number of service times refers to the maximum number of service times that each user, task, or request can obtain within a specific time period. By limiting the maximum number of service times, it is possible to effectively prevent certain users or applications from occupying too many system resources, thereby reducing the service quality of other users or tasks.

[0088] Service priority weight refers to the priority weight of different requests (or different users) when multiple requests compete for resources. Priority weight is usually related to the importance of the task, service level agreement or other business requirements. It determines which tasks or requests will be prioritized and which tasks will be delayed or queued when resources are insufficient.

[0089] The combination of these three, through reasonable configuration and dynamic adjustment, can ensure that the system runs efficiently and stably under high concurrent loads and provide appropriate service quality for each user or task.

[0090] The resource control device provided by the present invention is described below. The resource control device described below and the resource control method described above can be referenced to each other.

[0091] like Figure 4 As shown, the device comprises: A request receiving module 410, configured to receive an access request to a distributed file system; A control information determination module 420 is used to determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; The control and scheduling module 430 is used to determine the scheduling policy of the target user based on the control information of the target user, and perform quality of service QoS control on the access request based on the scheduling policy.

[0092] The resource control device provided by the present invention realizes the process of obtaining the control information of the access request by determining the target user in the client to which the access request belongs and obtaining the control information of the user corresponding to the access request pre-stored from the MAP during the process of executing the QoS control of the distributed file system. Based on the obtained control information, a scheduling strategy for the access request is generated to realize the scheduling control process of the access request, and only pays attention to the relationship between the access request and the user. Therefore, the method can constrain any request issued by the client user interacting with it, realizes the user-granular control process of all requests, and improves the control granularity of resources.

[0093] In one embodiment, the control and scheduling module 430 is specifically used to: The performing quality of service (QoS) control on the access request based on the scheduling strategy includes: The access request is added to the scheduling queue, and the scheduling policy is registered in the scheduling queue, so that after the consumer thread of the distributed file system obtains the access request from the scheduling queue, based on the registered scheduling policy, the pre-registered request processing function associated with the access request is executed according to the mClock algorithm to perform QoS control of the access request.

[0094] In one embodiment, the request receiving module 410 is specifically used to: After receiving the access request of the distributed file system, it also includes: In the case where the access request is an initial access request of the target user, determining user information of the target user based on parsing metadata of a client to which the access request belongs; The user information and the management and control information corresponding to the user information are added to the MAP.

[0095] In one embodiment, the control information determination module 420 is specifically used to: Before acquiring the control information for performing resource control on the target user from the metadata map MAP, the method further includes: It is determined that the access request is a request to enable the mClock algorithm to execute the QoS policy.

[0096] In one embodiment, the request receiving module 410 is further specifically configured to: After receiving the access request to the distributed file system, the method further includes: In the case where it is determined that the access request is not a request for enabling the mClock algorithm to execute the QoS policy, a processing function corresponding to the access request is called to execute a processing procedure of the access request.

[0097] In one embodiment, the request receiving module 410 is further specifically configured to: Before receiving the access request to the distributed file system, the method further includes: It is determined that a connection has been established with the client to which the access request belongs, and that authentication has been passed and a legal network session identifier has been obtained.

[0098] In one embodiment, the control and scheduling module 430 is further specifically used for: Determine that the management and control information at least includes a minimum service number, a maximum service number, and a service priority ratio.

[0099] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the resource management method, which includes: receiving an access request of a distributed file system; Determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Based on the control information of the target user, a scheduling policy of the target user is determined, and based on the scheduling policy, quality of service QoS control is performed on the access request.

[0100] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0101] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the resource management method provided by the above methods, the method includes: receiving an access request of a distributed file system; Determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Based on the control information of the target user, a scheduling policy of the target user is determined, and based on the scheduling policy, quality of service QoS control is performed on the access request.

[0102] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, the computer program being implemented when executed by a processor to execute the resource control method provided by the above methods, the method comprising: receiving an access request of a distributed file system; Determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Based on the control information of the target user, a scheduling policy of the target user is determined, and based on the scheduling policy, quality of service QoS control is performed on the access request.

[0103] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A resource management method, characterized in that: include: Receive access requests to the distributed file system; Determine a target user in the client to which the access request belongs, and obtain control information for resource control of the target user from a metadata map MAP, wherein the MAP includes control information of all users interacting with the distributed file system; Based on the control information of the target user, a scheduling policy of the target user is determined, and based on the scheduling policy, quality of service QoS control is performed on the access request.

2. The resource management method according to claim 1, characterized in that: Based on the scheduling strategy, the access request is subjected to quality of service (QoS) control, including: The access request is added to the scheduling queue, and the scheduling policy is registered in the scheduling queue, so that after the consumer thread of the distributed file system obtains the access request from the scheduling queue, based on the registered scheduling policy, the pre-registered request processing function associated with the access request is executed according to the mClock algorithm to perform QoS control of the access request.

3. The resource management method according to claim 1, characterized in that: After receiving the access request to the distributed file system, the method further includes: In the case where the access request is an initial access request of the target user, determining user information of the target user based on parsing metadata of a client to which the access request belongs; The user information and the management and control information corresponding to the user information are added to the MAP.

4. The resource management method according to claim 1, characterized in that: Before acquiring the control information for performing resource control on the target user from the metadata map MAP, the method further includes: It is determined that the access request is a request to enable the mClock algorithm to execute the QoS policy.

5. The resource management method according to claim 1, characterized in that: After receiving the access request to the distributed file system, the method further includes: In the case where it is determined that the access request is not a request for enabling the mClock algorithm to execute the QoS policy, a processing function corresponding to the access request is called to execute a processing procedure of the access request.

6. The resource management method according to claim 1, characterized in that: Before receiving the access request to the distributed file system, the method further includes: It is determined that a connection has been established with the client to which the access request belongs, and that authentication has been passed and a legal network session identifier has been obtained.

7. The resource management method according to any one of claims 1 to 6, characterized in that: The control information includes at least the minimum service times, the maximum service times and the service priority ratio.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the resource management method according to any one of claims 1 to 7 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the resource management and control method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the resource management and control method according to any one of claims 1 to 7 is implemented.

Images