Memory overconfiguration implementation method, system and equipment for multi-agent shared memory pool and medium

Abstract

The invention provides a memory overconfiguration implementation method, system and device for a multi-agent shared memory pool and a medium, and belongs to the technical field of memory resource allocation, the method comprises the following steps: classifying IO (Input / Output) in a storage system, and binding an agent node for each type of IO; configuring priority and weight parameters for the agent nodes bound with each type of IO according to a preset control page scheduling configuration strategy; for each agent node, limiting the speed of the IO of the corresponding category through a token bucket algorithm; and performing IO level marking on each agent node according to the use state of the control page of the IO of the corresponding category, and performing control page allocation scheduling on the IO of the corresponding category according to the IO level mark. The method not only can control the rate of the memory allocated by the memory pool, but also can realize refined allocation of the memory according to the business characteristics of the memory needing to be allocated, and is over-matched on the specified specification.

Description

Background Art

[0002] Memory over-provisioning occurs when the total memory size used by all businesses in a memory pool shared by multiple agents exceeds the total size of the memory pool.

[0003] With the rapid development of information technology, data storage systems are becoming increasingly important. They not only need to provide high-capacity and high-reliability data storage capabilities, but also need to be able to efficiently manage resources to cope with the growing demand for data processing. However, in data centers and cloud computing environments, storage systems often face the problem of resource constraints, especially in virtual machine environments, where memory over-provisioning is particularly common. Memory over-provisioning allows temporary expansion of memory resources through virtualization technology when physical memory is insufficient. For example, using swap space to expand memory solves the problem of memory over-provisioning. However, in Linux systems, swap technology is similar to Windows virtual memory. When the system memory is insufficient, temporarily unused data will be swapped to the swap partition of the disk to free up physical memory for other processes to use. When the process needs the data again, it is swapped back to physical memory from the swap partition. However, this method has the following significant disadvantages: the disk I / O performance is much lower than the memory, which leads to a slow data exchange process and poor overall system performance; the data swap cache process requires the participation of the CPU, which increases the CPU burden and leads to high CPU resource consumption; the size of the swap space needs to be planned in advance and is difficult to adjust dynamically according to actual needs, resulting in poor flexibility; when the swap space is exhausted, the operating system will trigger the OOM-killer mechanism and randomly kill the process that consumes the most memory, which may cause the machine to crash and has a high risk.

[0004] Therefore, in view of the above-mentioned defects, it is very necessary to provide a method, system, device and medium for implementing memory over-allocation of a multi-agent shared memory pool. Summary of the invention

[0005] In view of the above-mentioned shortcomings of solving the memory over-allocation problem by expanding memory through swap space, such as low performance, high CPU resource consumption, poor flexibility and high risk, the present invention provides a method, system, device and medium for implementing memory over-allocation of a multi-agent shared memory pool to solve the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for implementing memory over-allocation of a multi-agent shared memory pool, comprising the following steps:

[0007] S1. Classify the IO in the storage system and bind an agent node to each type of IO;

[0008] S2. Configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy;

[0009] S3. Use the token bucket algorithm to limit the speed of the corresponding category of IO for each proxy node;

[0010] S4. Mark each proxy node with an IO level according to the control page usage status of the corresponding category of IO, and then perform control page allocation scheduling for the corresponding category of IO according to the IO level mark.

[0011] Furthermore, the specific steps of step S1 are as follows:

[0012] S11. Classify the IO in the storage system according to source, front-end and back-end, or business type;

[0013] S12. Set a category ID for each type of IO, and bind a proxy node to each type of IO and its category ID. This allows the IO in the storage system to be accurately classified according to source, front-end, back-end, or business type, and assign a dedicated proxy node to each type of IO. This classification and binding mechanism helps the system better understand and respond to different types of IO requirements, and improves the flexibility and adaptability of the system.

[0014] Furthermore, the specific steps of step S2 are as follows:

[0015] S21. Get the preset control page scheduling configuration policy;

[0016] When the control page scheduling configuration strategy is the polling mode, go to step S22;

[0017] When the control page scheduling configuration policy is the priority mode, proceed to step S23;

[0018] When the control page scheduling configuration strategy is the priority + weight mode, proceed to step S24;

[0019] S22. Assign the same priority to each type of IO-bound proxy node, without assigning a weight parameter, and proceed to step S3;

[0020] S23. Obtain the importance of each type of IO-bound proxy node, and configure the priority of each type of IO-bound proxy node according to the importance, and proceed to step S3;

[0021] S24. Obtain the importance and pre-set weight parameters of each type of IO-bound proxy node, configure the priority for each type of IO-bound proxy node according to the importance, and allocate the corresponding type of IO according to the pre-set weight parameters. By configuring different priorities and weight parameters for each type of IO-bound proxy node, the system can flexibly adjust the priority and resource allocation of IO processing according to different control page scheduling configuration strategies (polling, priority, priority + weight). This dynamic adjustment mechanism helps the system ensure that IO requests for critical businesses are given priority in high-concurrency and high-load scenarios, thereby improving the overall performance and response speed of the system.

[0022] Furthermore, the specific steps of step S3 are as follows:

[0023] S31. Obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO;

[0024] S32. The proxy node calculates the timer period according to the IO speed limit value and adds a token to the token bucket according to the timer period;

[0025] S33. When the proxy node detects an IO request of a binding category, it checks whether there is an available token in the token bucket; if so, proceeds to step S34;

[0026] If not, proceed to step S35;

[0027] S34. The proxy node takes out a token from the token bucket and performs the IO operation corresponding to the IO request, and proceeds to step S4;

[0028] S35. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket, and returns to step S34. The application of the token bucket algorithm provides accurate rate limits for each type of IO request, avoiding system crashes or performance degradation caused by resource competition. By dynamically adjusting the capacity of the token bucket and the IO rate limit value, the system can flexibly respond to IO requirements under different load conditions, ensuring the stability and predictability of the system. Further, the specific steps of step S35 are as follows:

[0029] S351. The proxy node pre-generates a request queue;

[0030] S352. The proxy node puts the IO request into the request queue;

[0031] S353. The proxy node checks whether there are available tokens in the token bucket;

[0032] If yes, go to step S354;

[0033] If not, proceed to step S355;

[0034] S354. The proxy node allocates the available token to the IO request that first enters the request queue, and deletes the IO request from the request queue;

[0035] S355. The proxy node returns to step S34 to perform the IO operation while checking whether the request queue is empty;

[0036] If yes, return to step S33;

[0037] If not, return to step S353. Based on the token bucket algorithm, the request queue mechanism is introduced to effectively alleviate the IO request waiting problem caused by insufficient tokens. By putting the waiting IO requests into the request queue and sorting and scheduling them according to certain strategies such as first-in-first-out, the system can ensure that all IO requests can be processed in a timely manner, while avoiding request loss or delay caused by resource competition.

[0038] Furthermore, the specific steps of step S4 are as follows:

[0039] S41. Obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node;

[0040] S42. The proxy node directly allocates control pages to IO categories with IO level marked as level 1, allocates over-allocated control pages to IO categories with IO level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and re-acquires the IO control page usage status after waiting for a set time period for IO categories with IO level 3. Based on the IO level marking and over-allocated control page allocation strategy of the IO control page usage status, the system can dynamically adjust the allocation and scheduling of control pages according to different IO requirements and resource conditions. By allocating control pages of different numbers and priorities to IO categories of different levels, the system can ensure that IO requests for key businesses are given priority, while making rational use of over-allocated resources to improve the overall performance and resource utilization of the system.

[0041] Furthermore, the specific steps of step S41 are as follows:

[0042] S411. Obtaining the relative position relationship of the number of various types of control pages for each category of IO, wherein the categories of the number of control pages include the number of control pages, the number of used control pages, the number of committed control pages, and the number of over-allocated control pages;

[0043] When the sum of the number of free control pages and the number of used control pages is less than or equal to the number of committed control pages, proceed to step S412;

[0044] When the sum of the number of free control pages and the number of used control pages is greater than the number of committed control pages, and less than or equal to the sum of the number of committed control pages and the number of over-allocated control pages, proceed to step S413;

[0045] When the sum of the number of free control pages and the number of used control pages is greater than the sum of the number of committed control pages and the number of over-allocated control pages, the process proceeds to step S414;

[0046] S412. Mark the IO of this category as level 1 and mark it with green, and proceed to step S42;

[0047] S413. Mark the IO of this category as secondary and mark it with yellow, and proceed to step S42;

[0048] S414. Mark the IO of this category as level 3 and mark it in red;

[0049] In step S42, control pages are allocated for the IO category with IO level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters. The specific steps are as follows:

[0050] SS1. Detection and control of paging configuration policy;

[0051] When the control page scheduling configuration policy is the polling mode, enter step SS2;

[0052] When the control page scheduling configuration policy is the priority mode, go to step SS3;

[0053] When the control page scheduling configuration strategy is the priority + weight mode, go to step SS4;

[0054] SS2. Allocate over-allocated control pages to each type of IO in turn, and then end;

[0055] SS3. Allocate over-allocated control pages to each type of IO according to priority, and then end;

[0056] SS4. Allocate over-allocated control pages for each type of IO according to the priority and weight parameters. Steps S411-S414 provide detailed determination basis for the level marking of IO categories. By comparing the relative relationship between the number of free control pages, the number of used control pages, the number of committed control pages, and the number of over-allocated control pages, the system can accurately set the corresponding level marking for each type of IO. This refined level division helps the system better understand and respond to different types of IO requirements, and improves the flexibility and response speed of the system.

[0057] In a second aspect, the present invention provides a memory over-provisioning implementation system for a multi-agent shared memory pool, including: an IO category and agent node binding module, which is used to classify IOs in a storage system and bind an agent node to each type of IO;

[0058] The priority and weight configuration module is used to configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy;

[0059] The IO speed limit module is used to limit the speed of the corresponding category of IO for each proxy node through the token bucket algorithm; the IO marking and over-allocation scheduling module is used to mark the IO level of each proxy node according to the control page usage status of the corresponding category of IO, and then allocate and schedule the control page for the corresponding category of IO according to the IO level mark.

[0060] Furthermore, the IO category and proxy node binding module include:

[0061] The IO classification unit is used to classify the IO in the storage system according to the source, foreground and background or business type; the IO binding unit is used to set a category ID for each type of IO and bind an agent node to each type of IO and its category ID.

[0062] Furthermore, the priority and weight configuration module includes:

[0063] A scheduling configuration strategy acquisition unit, used to acquire a preset control page scheduling configuration strategy;

[0064] A polling strategy configuration unit, used to assign the same priority to each type of IO-bound proxy node when the control page scheduling configuration strategy is a polling mode, without assigning a weight parameter;

[0065] The priority policy configuration unit is used to obtain the importance of each type of IO-bound proxy node when the control page scheduling configuration policy is the priority mode, and configure the priority of each type of IO-bound proxy node according to the importance;

[0066] The priority weight strategy configuration unit is used to obtain the importance and pre-set weight parameters of the proxy nodes bound to each type of IO when the control page scheduling configuration strategy is the priority + weight mode, and configure the priority for the proxy nodes bound to each type of IO according to the importance, and allocate the IO of the corresponding category according to the pre-set weight parameters.

[0067] Furthermore, the IO rate limiting module includes:

[0068] The IO speed limit and token bucket configuration unit is used to obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO;

[0069] A token adding unit is used for the proxy node to calculate the timer period according to the IO speed limit value and add a token to the token bucket according to the timer period;

[0070] The token checking unit is used to check whether there is an available token in the token bucket when the proxy node detects an IO request of the binding category;

[0071] The token extraction execution unit is used for the proxy node to extract a token from the token bucket when there are available tokens in the token bucket and execute the IO operation corresponding to the IO request;

[0072] The token waiting unit is used when there are no available tokens in the token bucket. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket.

[0073] Furthermore, the IO marking and over-allocation scheduling module includes:

[0074] An IO marking unit is used to obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of the IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node;

[0075] The control page allocation unit is used to directly allocate control pages for IO categories marked as level 1 by the proxy node, to perform over-allocation of control pages for IO categories marked as level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and to re-acquire the IO control page usage status for IO categories marked as level 3 after waiting for a set time period.

[0076] In a third aspect, the present invention provides an electronic device, including a processor and a memory;

[0077] The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in the first aspect above.

[0078] In a fourth aspect, the present invention provides a storage medium,

[0079] The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in the first aspect.

[0080] The beneficial effects of the present invention are:

[0081] The method, system, device and medium for implementing memory over-allocation of a proxy shared memory pool provided by the present invention can not only control the rate of memory allocated by the memory pool, but also allocate memory according to business characteristics as required, thereby realizing refined allocation of memory and over-allocation on specified specifications.

[0082] In addition, the invention has a reliable design principle, a simple structure and a very broad application prospect.

[0083] It can be seen that compared with the prior art, the present invention has outstanding substantive features and significant progress, and the beneficial effects of its implementation are also obvious. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0085] Figure 1 It is a flow chart of a method for implementing memory over-allocation of a multi-agent shared memory pool according to the present invention.

[0086] Figure 2 It is a schematic diagram of a memory over-allocation implementation system of a multi-agent shared memory pool of the present invention. DETAILED DESCRIPTION

[0087] The memory over-allocation implementation method of a multi-agent shared memory pool can be applied to one or more devices, wherein the device is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0088] An electronic device can be any electronic product that can interact with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), an interactive network television (IPTV), etc. The network in which the electronic device is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0089] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0090] Embodiment 1:

[0091] See also Figure 1 The figure shows a method for implementing memory over-allocation of a multi-agent shared memory pool in a specific implementation, comprising the following steps:

[0092] S1. Classify the IO in the storage system and bind an agent node to each type of IO;

[0093] S2. Configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy;

[0094] S3. Use the token bucket algorithm to limit the speed of the corresponding category of IO for each proxy node;

[0095] S4. Mark each proxy node with an IO level according to the control page usage status of the corresponding category of IO, and then perform control page allocation scheduling for the corresponding category of IO according to the IO level mark.

[0096] The memory over-provisioning implementation method of the multi-agent shared memory pool of the present invention significantly improves the resource utilization and IO processing efficiency of the storage system by introducing the multi-agent shared memory pool and memory over-provisioning technology; by classifying IO and binding the proxy nodes, the refined management and optimized scheduling of IO requests are realized; at the same time, the IO level marking and over-provisioning control page allocation strategy based on the control page usage status ensure the stability and performance of the system under high load.

[0097] Embodiment 2:

[0098] Different from Example 1, the specific steps of step S1 are as follows:

[0099] S11. Classify the IO in the storage system according to source, front-end and back-end, or business type;

[0100] For example, if we classify the IO in the storage system according to the front-end and back-end IO, the results are shown in Table 1 below:

[0101] Table 1

[0102] Classification Category ID Front desk IO 0 Background IO 1

[0103] S12. Set a category ID for each type of IO, and bind a proxy node to each type of IO and its category ID; the binding results are shown in Table 2 below:

[0104] Table 2

[0105] Classification Category ID Bound Agent Front desk IO 0 Agent_0 Background IO 1 Agent_1

[0106] The specific steps of step S2 are as follows:

[0107] S21. Get the preset control page scheduling configuration policy;

[0108] When the control page scheduling configuration strategy is the polling mode, go to step S22;

[0109] When the control page scheduling configuration policy is the priority mode, proceed to step S23;

[0110] When the control page scheduling configuration strategy is the priority + weight mode, proceed to step S24;

[0111] S22. Assign the same priority to each type of IO-bound proxy node, without assigning a weight parameter, and proceed to step S3;

[0112] For example, as shown in Table 3, the proxy node priorities are assigned as follows:

[0113] Table 3

[0114] Classification Category ID Bound Agent Priority Front desk IO 0 Agent_0 0 Background IO 1 Agent_1 0

[0115] S23. Obtain the importance of each type of IO-bound proxy node, and configure the priority of each type of IO-bound proxy node according to the importance, and proceed to step S3;

[0116] For example, Table 4 assigns proxy node priorities as follows:

[0117] Table 4

[0118] Classification Category ID Bound Agent Priority Front Desk IO 0 Agent_0 0 (High) Background IO 1 Agent_1 1 (low)

[0119] S24. Obtain the importance and pre-set weight parameters of the proxy nodes bound to each type of IO, configure the priority of the proxy nodes bound to each type of IO according to the importance, and allocate the IOs of the corresponding types according to the pre-set weight parameters;

[0120] For example, Table 5 assigns proxy node priorities as follows:

[0121] Table 5

[0122] Classification Category ID Bound Agent Priority Weight Front Desk IO 0 Agent_0 0 (High) 10 Background IO 1 Agent_1 1 (low) 20

[0123] The specific steps of step S3 are as follows:

[0124] S31. Obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO;

[0125] S32. The proxy node calculates the timer period according to the IO speed limit value, and adds a token to the token bucket according to the timer period. For example, assuming that the IO speed limit is 10000 IOPS and the token bucket size is 100, we can calculate that the timer period is 100us according to the IO speed limit, that is, a token is put into the token bucket every 100us. The token bucket is set to 100, which means that a maximum of 100 IOs can be burst instantly.

[0126] S33. When the proxy node detects an IO request of a binding category, it checks whether there is an available token in the token bucket; if so, proceeds to step S34;

[0127] If not, proceed to step S35;

[0128] S34. The proxy node takes out a token from the token bucket and performs the IO operation corresponding to the IO request, and proceeds to step S4;

[0129] S35. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket, and returns to step S34;

[0130] The specific steps of step S4 are as follows:

[0131] S41. Obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node;

[0132] S42. The proxy node directly allocates control pages to IO categories marked as level 1, and over-allocates control pages to IO categories marked as level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and waits for a set time period for IO categories marked as level 3 before re-obtaining the IO control page usage status.

[0133] Embodiment 3:

[0134] Different from Example 1, the specific steps of step S1 are as follows:

[0135] S11. Classify the IO in the storage system according to source, front-end and back-end, or business type;

[0136] S12. Set a category ID for each type of IO, and bind an agent node to each type of IO and its category ID; the specific steps of step S2 are as follows:

[0137] S21. Get the preset control page scheduling configuration policy;

[0138] When the control page scheduling configuration strategy is the polling mode, go to step S22;

[0139] When the control page scheduling configuration policy is the priority mode, proceed to step S23;

[0140] When the control page scheduling configuration strategy is the priority + weight mode, proceed to step S24;

[0141] S22. Assign the same priority to each type of IO-bound proxy node, without assigning a weight parameter, and proceed to step S3;

[0142] S23. Obtain the importance of each type of IO-bound proxy node, and configure the priority of each type of IO-bound proxy node according to the importance, and proceed to step S3;

[0143] S24. Obtain the importance and pre-set weight parameters of the proxy nodes bound to each type of IO, configure the priority of the proxy nodes bound to each type of IO according to the importance, and allocate the IOs of the corresponding types according to the pre-set weight parameters;

[0144] The specific steps of step S3 are as follows:

[0145] S31. Obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO;

[0146] S32. The proxy node calculates the timer period according to the IO speed limit value and adds a token to the token bucket according to the timer period;

[0147] S33. When the proxy node detects an IO request of a binding category, it checks whether there is an available token in the token bucket; if so, proceeds to step S34;

[0148] If not, proceed to step S35;

[0149] S34. The proxy node takes out a token from the token bucket and performs the IO operation corresponding to the IO request, and proceeds to step S4;

[0150] S35. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket, and returns to step S34; the specific steps of step S35 are as follows:

[0151] S351. The proxy node pre-generates a request queue;

[0152] S352. The proxy node puts the IO request into the request queue;

[0153] S353. The proxy node checks whether there are available tokens in the token bucket;

[0154] If yes, go to step S354;

[0155] If not, proceed to step S355;

[0156] S354. The proxy node allocates the available token to the IO request that first enters the request queue, and deletes the IO request from the request queue;

[0157] S355. The proxy node returns to step S34 to perform the IO operation while checking whether the request queue is empty;

[0158] If yes, return to step S33;

[0159] If not, return to step S353;

[0160] The specific steps of step S4 are as follows:

[0161] S41. Obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node; the specific steps of step S41 are as follows:

[0162] S411. Obtaining the relative position relationship of the number of various types of control pages for each category of IO, wherein the categories of the number of control pages include the number of control pages, the number of used control pages, the number of committed control pages, and the number of over-allocated control pages;

[0163] When the sum of the number of free control pages and the number of used control pages is less than or equal to the number of committed control pages, proceed to step S412;

[0164] When the sum of the number of free control pages and the number of used control pages is greater than the number of committed control pages, and less than or equal to the sum of the number of committed control pages and the number of over-allocated control pages, proceed to step S413;

[0165] When the sum of the number of free control pages and the number of used control pages is greater than the sum of the number of committed control pages and the number of over-allocated control pages, the process proceeds to step S414;

[0166] S412. Mark the IO of this category as level 1 and mark it with green, and proceed to step S42;

[0167] S413. Mark the IO of this category as secondary and mark it with yellow, and proceed to step S42;

[0168] S414. Mark the IO of this category as level 3 and mark it in red;

[0169] S42. The proxy node directly allocates control pages for IO categories marked as level 1, allocates over-allocated control pages for IO categories marked as level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and re-acquires the IO control page usage status for IO categories marked as level 3 after waiting for a set period of time;

[0170] In step S42, control pages are allocated for the IO category with IO level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters. The specific steps are as follows:

[0171] SS1. Detection and control of paging configuration policy;

[0172] When the control page scheduling configuration policy is the polling mode, enter step SS2;

[0173] When the control page scheduling configuration policy is the priority mode, go to step SS3;

[0174] When the control page scheduling configuration strategy is the priority + weight mode, go to step SS4;

[0175] SS2. Allocate over-allocated control pages to each type of IO in turn, and then end;

[0176] SS3. Allocate over-allocated control pages to each type of IO according to priority, and then end;

[0177] SS4. Allocate over-allocated control pages for each type of IO according to priority and weight parameters;

[0178] For example, IO is marked by IO staining to dye IO into one of green, yellow, and red. The specific staining scheme is as follows:

[0179] When the number of free CPs + the number of used CPs is less than or equal to COMMIT_CP_NUM, the IO is marked green;

[0180] When the number of free CPs + the number of used CPs is greater than COMMIT_CP_NUM and less than or equal to COMMIT_CP_NUM + EXCEED_CP_NUM, the IO is marked yellow;

[0181] When the number of free CPs + the number of used CPs is greater than COMMIT_CP_NUM + EXCEED_CP_NUM, the IO is marked red;

[0182] For green IO, CP is directly allocated;

[0183] For yellow IO, when the configured scheduling policy is RR, over-allocated CPs are allocated to each type of IO in turn; when the configured scheduling policy is SP, over-allocated CPs are allocated to IO in descending order of priority; when the configured scheduling policy is SP+Wight, over-allocated CPs are allocated to each type of IO according to priority+weight.

[0184] For red IO, simply delay for a while and try again from the beginning.

[0185] It should be understood that the order of execution of the steps in the above embodiment does not mean that the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0186] The following is an embodiment of a system for implementing memory over-provisioning of a multi-agent shared memory pool provided by an embodiment of the present disclosure. The device and the method for implementing memory over-provisioning of a multi-agent shared memory pool in the above-mentioned embodiments belong to the same inventive concept. For details not fully described in the embodiment of the system for implementing memory over-provisioning of a multi-agent shared memory pool, reference may be made to the embodiment of the method for implementing memory over-provisioning of a multi-agent shared memory pool in the above-mentioned embodiments.

[0187] like Figure 2 As shown, the system includes:

[0188] The IO category and proxy node binding module is used to classify the IO in the storage system and bind an agent node to each type of IO;

[0189] The priority and weight configuration module is used to configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy;

[0190] The IO speed limit module is used to limit the speed of the corresponding category of IO for each proxy node through the token bucket algorithm; the IO marking and over-allocation scheduling module is used to mark the IO level of each proxy node according to the control page usage status of the corresponding category of IO, and then allocate and schedule the control page for the corresponding category of IO according to the IO level mark.

[0191] In this embodiment, the IO category and proxy node binding module includes:

[0192] The IO classification unit is used to classify the IO in the storage system according to the source, front-end and back-end, or business type; the IO binding unit is used to set a category ID for each type of IO and bind an agent node to each type of IO and its category ID;

[0193] The priority and weight configuration modules include:

[0194] A scheduling configuration strategy acquisition unit, used to acquire a preset control page scheduling configuration strategy;

[0195] A polling strategy configuration unit, used to assign the same priority to each type of IO-bound proxy node when the control page scheduling configuration strategy is a polling mode, without assigning a weight parameter;

[0196] The priority policy configuration unit is used to obtain the importance of each type of IO-bound proxy node when the control page scheduling configuration policy is the priority mode, and configure the priority of each type of IO-bound proxy node according to the importance;

[0197] The priority weight strategy configuration unit is used to obtain the importance and pre-set weight parameters of the proxy nodes bound to each type of IO when the control page scheduling configuration strategy is the priority + weight mode, and configure the priority for the proxy nodes bound to each type of IO according to the importance, and allocate the IO of the corresponding category according to the pre-set weight parameters;

[0198] The IO rate limit module includes:

[0199] The IO speed limit and token bucket configuration unit is used to obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO;

[0200] A token adding unit is used for the proxy node to calculate the timer period according to the IO speed limit value and add a token to the token bucket according to the timer period;

[0201] The token checking unit is used to check whether there is an available token in the token bucket when the proxy node detects an IO request of the binding category;

[0202] The token extraction execution unit is used for the proxy node to extract a token from the token bucket when there are available tokens in the token bucket and execute the IO operation corresponding to the IO request;

[0203] The token waiting unit is used when there are no available tokens in the token bucket. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket.

[0204] The IO marking and over-allocation scheduling module includes:

[0205] An IO marking unit is used to obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of the IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node;

[0206] The control page allocation unit is used to directly allocate control pages for IO categories marked as level 1 by the proxy node, to perform over-allocation of control pages for IO categories marked as level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and to re-acquire the IO control page usage status for IO categories marked as level 3 after waiting for a set time period.

[0207] The memory over-allocation implementation method of the multi-agent shared memory pool provided by the present invention is a combination of the modules and algorithm steps of each example described in the embodiments disclosed herein, and can be implemented by electronic hardware, computer software or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0208] Those skilled in the art will appreciate that various aspects of the method for implementing memory over-allocation of a multi-agent shared memory pool provided by the present invention can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software, which can be collectively referred to as a "circuit", "module" or "system" here.

[0209] In the non-transitory computer-readable storage medium storing the memory over-allocation implementation method of the multi-agent shared memory pool, it can be, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0210] Although the present invention has been described in detail with reference to the accompanying drawings and in combination with the preferred embodiments, the present invention is not limited thereto. Without departing from the spirit and essence of the present invention, a person of ordinary skill in the art may make various equivalent modifications or substitutions to the embodiments of the present invention, and these modifications or substitutions shall be within the scope of the present invention. Any person of ordinary skill in the art may easily think of changes or substitutions within the technical scope disclosed by the present invention, and these shall be within the scope of protection of the present invention.

Claims

1. A method for implementing memory over-allocation of a multi-agent shared memory pool, characterized in that: The steps include: S1. Classify the IO in the storage system and bind an agent node to each type of IO; S2. Configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy; S3. Use the token bucket algorithm to limit the speed of the corresponding category of IO for each proxy node; S4. Mark each proxy node with an IO level according to the control page usage status of the corresponding category of IO, and then perform control page allocation scheduling for the corresponding category of IO according to the IO level mark.

2. The method for implementing memory over-allocation of a multi-agent shared memory pool according to claim 1, characterized in that: The specific steps of step S1 are as follows: S11. Classify the IO in the storage system according to source, front-end and back-end, or business type; S12. Set a category ID for each type of IO, and bind an agent node to each type of IO and its category ID.

3. The method for implementing memory over-allocation of a multi-agent shared memory pool according to claim 1, characterized in that: The specific steps of step S2 are as follows: S21. Get the preset control page scheduling configuration policy; When the control page scheduling configuration strategy is the polling mode, go to step S22; When the control page scheduling configuration policy is the priority mode, proceed to step S23; When the control page scheduling configuration strategy is the priority + weight mode, proceed to step S24; S22. Assign the same priority to each type of IO-bound proxy node, without assigning a weight parameter, and proceed to step S3; S23. Obtain the importance of each type of IO-bound proxy node, and configure the priority of each type of IO-bound proxy node according to the importance, and proceed to step S3; S24. Obtain the importance and pre-set weight parameters of the proxy nodes bound to each type of IO, configure the priority of the proxy nodes bound to each type of IO according to the importance, and allocate the IOs of the corresponding types according to the pre-set weight parameters.

4. The method for implementing memory over-allocation of a multi-agent shared memory pool according to claim 1, characterized in that: The specific steps of step S3 are as follows: S31. Obtain the IO speed limit value and token bucket capacity pre-set for each type of IO, and configure them in the proxy node bound to this type of IO; S32. The proxy node calculates the timer period according to the IO speed limit value and adds a token to the token bucket according to the timer period; S33. When the proxy node detects an IO request of a binding category, it checks whether there is an available token in the token bucket; If yes, go to step S34; If not, proceed to step S35; S34. The proxy node takes out a token from the token bucket and performs the IO operation corresponding to the IO request, and proceeds to step S4; S35. The proxy node puts the IO request into the request queue until there are enough tokens in the token bucket, and returns to step S34.

5. The method for implementing memory over-allocation of a multi-agent shared memory pool as claimed in claim 4, characterized in that: The specific steps of step S35 are as follows: S351. The proxy node pre-generates a request queue; S352. The proxy node puts the IO request into the request queue; S353. The proxy node checks whether there are available tokens in the token bucket; If yes, go to step S354; If not, proceed to step S355; S354. The proxy node allocates the available token to the IO request that first enters the request queue, and deletes the IO request from the request queue; S355. The proxy node returns to step S34 to perform the IO operation while checking whether the request queue is empty; If yes, return to step S33; If not, return to step S353.

6. The method for implementing memory over-allocation of a multi-agent shared memory pool according to claim 1, characterized in that: The specific steps of step S4 are as follows: S41. Obtain the control page usage status of each category of IO, and set the IO category to three IO levels according to the control page usage status of IO, perform corresponding IO level marking, and configure the IO level marking to the corresponding bound proxy node; S42. The proxy node directly allocates control pages to IO categories marked as level 1, and over-allocates control pages to IO categories marked as level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters, and waits for a set time period for IO categories marked as level 3 before re-obtaining the IO control page usage status.

7. The method for implementing memory over-allocation of a multi-agent shared memory pool according to claim 1, characterized in that: The specific steps of step S41 are as follows: S411. Obtaining the relative position relationship of the number of various types of control pages for each category of IO, wherein the categories of the number of control pages include the number of control pages, the number of used control pages, the number of committed control pages, and the number of over-allocated control pages; When the sum of the number of free control pages and the number of used control pages is less than or equal to the number of committed control pages, proceed to step S412; When the sum of the number of free control pages and the number of used control pages is greater than the number of committed control pages, and less than or equal to the sum of the number of committed control pages and the number of over-allocated control pages, proceed to step S413; When the sum of the number of free control pages and the number of used control pages is greater than the sum of the number of committed control pages and the number of over-allocated control pages, the process proceeds to step S414; S412. Mark the IO of this category as level 1 and mark it with green, and proceed to step S42; S413. Mark the IO of this category as secondary and mark it with yellow, and proceed to step S42; S414. Mark this category of IO as level 3 and use red as the mark; In step S42, control pages are allocated for the IO category with IO level 2 according to the control page scheduling configuration strategy and the set priority and weight parameters. The specific steps are as follows: SS1. Detection and control of paging configuration policy; When the control page scheduling configuration policy is the polling mode, enter step SS2; When the control page scheduling configuration policy is the priority mode, go to step SS3; When the control page scheduling configuration strategy is the priority + weight mode, go to step SS4; SS2. Allocate over-allocated control pages to each type of IO in turn, and then end; SS3. Allocate over-allocated control pages to each type of IO according to priority, and then end; SS4. Allocate over-allocated control pages to each type of IO according to priority and weight parameters.

8. A memory over-allocation implementation system for a multi-agent shared memory pool, characterized in that: include: The IO category and proxy node binding module is used to classify the IO in the storage system and bind an agent node to each type of IO; The priority and weight configuration module is used to configure the priority and weight parameters for each type of IO-bound proxy node according to the preset control page scheduling configuration strategy; The IO speed limit module is used to limit the speed of the corresponding category of IO for each proxy node through the token bucket algorithm; the IO marking and over-allocation scheduling module is used to mark the IO level of each proxy node according to the control page usage status of the corresponding category of IO, and then allocate and schedule the control page for the corresponding category of IO according to the IO level mark.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that the electronic device executes the method described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method described in any one of claims 1 to 7.

Images