Distributed storage RPC flow control method and system based on adaptive dynamic token

Abstract

The present invention discloses a distributed storage RPC flow control method and system based on adaptive dynamic tokens, and relates to the field of distributed storage. The steps of the method include: creating an RPC subqueue corresponding to each target that sends and receives RPCs on the local end, and allocating available tokens to each RPC subqueue according to the memory of the local end; after the local end sends RPCs according to the available tokens, the number of available tokens of the RPC subqueue is updated according to the overall token holding time of each RPC subqueue. The present invention uses flexible differentiated RPC flow control to enable targets that can withstand greater pressure to obtain more available tokens, thereby maximizing memory utilization efficiency under limited memory constraints; at the same time, the present invention can also dynamically and adaptively adjust the number of available tokens of each RPC subqueue as the cluster size changes. Therefore, the present invention can make full and reasonable use of limited resources and has strong flexibility.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular to a distributed storage RPC flow control method and system based on adaptive dynamic tokens. Background Art

[0002] As the cornerstone of today's cloud computing, distributed storage has extremely high requirements for cluster network communication performance. Therefore, remote procedure calls (RPCs) are commonly used to achieve communication between nodes in the cluster. Under limited memory constraints, the memory resources that RPC can use have to be limited to a certain amount, so RPC flow control must be performed. The usual practice is to use engine-granular static RPC flow control, that is, within a certain period of time, the number of unreplied RPCs sent by the working threads (including control-plane threads and IO-plane threads, hereinafter referred to as targets) in one storage engine (hereinafter referred to as the local end) to other storage engines (hereinafter referred to as the peer end) is limited to a certain number of tokens. RPCs exceeding the number of tokens will be placed in the flow control queue waiting.

[0003] See Figure 1 As shown, all RPCs on this end are stored in the RPC queue, and available tokens are allocated for the RPCs sent to the target of each peer (Engine1~3 in the figure); before sending, it is determined whether there are available tokens to the destination engine. If so, it is sent directly and holds a token; otherwise, it enters the RPC flow control queue and waits. When an RPC callback releases the token, the RPCs in the RPC flow control queue obtain the tokens in turn and are sent.

[0004] The drawback of this method is that when the cluster size changes (for example, increasing or decreasing the storage engine), it will lead to unreasonable resource utilization and poor flexibility. Specifically, when the cluster size decreases, memory resources may not be fully utilized, and when the cluster size increases, it may lead to excessive resource usage. Summary of the invention

[0005] In view of the defects in the prior art, the technical problem solved by the present invention is: how to match the number of RPCs sent from the local end to the target of the other end with the network resource utilization rate of the target of the other end; thereby rationally utilizing resources and improving flexible performance.

[0006] To achieve the above objectives, in the first aspect, an embodiment of the present application provides a distributed storage RPC flow control method based on adaptive dynamic tokens, comprising the following steps: creating an RPC sub-queue on the local end for each target of the peer end for sending and receiving RPCs, and allocating available tokens to each RPC sub-queue according to the local end's memory; after the local end sends the RPC based on the available tokens, the number of available tokens of each RPC sub-queue is updated according to the overall token holding time of each RPC sub-queue.

[0007] In combination with the first aspect, in one embodiment, before sending the RPC according to the available tokens, the following steps are also included: setting a token minimum limit for the number of available tokens for each RPC subqueue, and the minimum number of available tokens for each RPC subqueue is above the token minimum limit.

[0008] In combination with the first aspect, in one embodiment, the process of updating the number of available tokens of each RPC subqueue according to the overall token holding time of each RPC subqueue includes: calculating the overall token holding time ∑t of each RPC subqueue k within the update period T k , the calculation formula is: where t now Represents the current moment, Delegate RPC i The moment to get a valid token and send an RPC, Represents the RPC i The moment when the token is released after the callback; according to ∑t k Calculate the number of available tokens N that need to be reallocated for the current RPC subqueue k , the calculation formula is: Where N represents the number of available tokens on the local end. min represents the minimum token limit of the RPC subqueue, K represents the total number of all RPC subqueues, ∑ K ∑t k Represents the sum of the overall token holding time of each RPC subqueue.

[0009] In combination with the first aspect, in one embodiment, the method further includes the following steps: when each RPC callback is made, the available token holding time of the RPC in the next update cycle T is calculated, and added to the overall token holding time of the RPC subqueue to which it belongs.

[0010] In combination with the first aspect, in one embodiment, it is characterized in that the method also includes the following steps: when a pool map update is monitored, the update cycle is shortened and the cycle starting point is refreshed according to the increase or decrease of the RPC subqueue corresponding to the update of the pool map, and the subsequent update cycle is restored to the value before the pool map update.

[0011] In combination with the first aspect, in one embodiment, an RPC sub-queue is created for each target of the other end for sending and receiving RPCs on the local end, and the process of allocating available tokens to each RPC sub-queue according to the memory of the local end includes: creating an RPC sub-queue for each target of the other end, and the initial number of available tokens of the RPC sub-queue is the total number of available tokens that can be obtained in the memory of the local end, divided by the number of RPC sub-queues.

[0012] In combination with the first aspect, in one embodiment, the process of sending RPC by this end according to the available token includes: determining whether there is any available token in the current RPC sub-queue before sending, if so, sending RPC and holding an available token, otherwise waiting; the RPC callback of the current RPC sub-queue will release the available token and return it to the current RPC sub-queue.

[0013] In a second aspect, an embodiment of the present application provides a storage medium having a computer program stored thereon, wherein the computer program implements the above method when executed by a processor.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that runs on the processor, and the processor implements the above method when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a distributed storage RPC flow control system based on adaptive dynamic tokens, including an RPC subqueue creation module, an available token allocation module, an RPC sending module, and an available token update module arranged at the local end;

[0016] The RPC subqueue creation module is used to: implement the above-mentioned process of creating an RPC subqueue for each target of sending and receiving RPCs; and set the minimum token limit for the number of available tokens for each RPC subqueue;

[0017] The available token allocation module is used to: implement the above-mentioned process of allocating available tokens to each RPC sub-queue according to the memory of the local end;

[0018] The RPC sending module is used to: implement the above-mentioned process of sending RPC according to the available token;

[0019] The available token update module is used to implement the above-mentioned process of updating the number of available tokens of each RPC sub-queue according to the overall token holding time of the RPC sub-queue.

[0020] Compared with the prior art, the advantages of the present invention are:

[0021] (1) The present invention refines the RPC subqueue allocation granularity, and on the basis of allocating available tokens to each RPC subqueue at the target granularity, can update the number of available tokens of the RPC subqueue according to the overall holding time of the tokens of each RPC subqueue. On this basis, when the overall holding time of the subqueue tokens is longer, it means that the work pressure of the target is greater, and at this time, the available tokens of the corresponding RPC subqueue are increased; when the overall holding time of the subqueue tokens is shorter, it means that the work pressure of the target is less, and at this time, the available tokens of the corresponding RPC subqueue are reduced.

[0022] In view of this, the present invention uses flexible differentiated RPC flow control to enable targets that can withstand greater pressure to obtain more available tokens, thereby maximizing memory utilization efficiency under limited memory constraints; at the same time, the present invention can also dynamically and adaptively adjust the number of available tokens for each RPC subqueue as the cluster scale changes. Therefore, the present invention can fully and reasonably utilize limited resources and has strong flexibility.

[0023] (2) The present invention can achieve: RPC subqueues in a relatively busy or waiting state will obtain newly added available tokens, so that they can send RPCs; while RPC subqueues in a relatively idle state will have the number of available tokens reduced. Therefore, the present invention can dynamically evaluate the working pressure of the peer target through the total holding time of the available tokens of the RPC subqueues, and then dynamically update the token allocation weight for each peer target, adaptively adjust the number of available tokens according to the peer pressure state, and make up for the busy by taking advantage of the idle, so as to maximize the system performance based on limited resources.

[0024] (3) The present invention sets a minimum limit of available tokens for each RPC subqueue, so that even if the number of available tokens is reduced in a relatively idle RPC subqueue, the minimum available tokens can always be guaranteed to be available, and new RPC sending requests will not be blocked. Even if some targets have a large number of RPC timeouts due to abnormalities, and the overall holding time of available tokens is relatively long, it will not affect the RPC sent to other targets too much, and fault isolation between multiple data pools in the storage cluster can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0026] Figure 1It is a schematic diagram of the engine granularity static RPC flow control mechanism in the prior art;

[0027] Figure 2 Schematic diagram of calculation of the total holding time of available tokens of all RPC subqueues within a time window in an embodiment of the present invention;

[0028] Figure 3 It is a flow chart of a distributed storage RPC flow control method based on adaptive dynamic tokens in an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of the target-granularity dynamic RPC flow control mechanism in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0031] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0032] The distributed storage RPC flow control method based on adaptive dynamic tokens in the embodiment of the present invention includes the following steps: creating an RPC sub-queue at the local end for each target of the peer end that sends and receives RPC, and allocating available tokens to each RPC sub-queue according to the local end's memory; after the local end sends the RPC according to the available tokens, the number of available tokens of each RPC sub-queue is updated according to the overall token holding time of each RPC sub-queue.

[0033] It can be seen that the present invention refines the RPC subqueue allocation granularity, and on the basis of allocating available tokens to each RPC subqueue at the target granularity, can update the number of available tokens of the RPC subqueue according to the overall holding time of the tokens of each RPC subqueue. On this basis, when the overall holding time of the subqueue token is longer, it means that the work pressure of the target is greater, and the available tokens of the corresponding RPC subqueue are increased at this time; when the overall holding time of the subqueue token is shorter, it means that the work pressure of the target is less, and the available tokens of the corresponding RPC subqueue are reduced at this time.

[0034] In view of this, the present invention uses flexible differentiated RPC flow control to enable targets that can withstand greater pressure to obtain more available tokens, thereby maximizing memory utilization efficiency under limited memory constraints; at the same time, the present invention can also dynamically and adaptively adjust the number of available tokens for each RPC subqueue as the cluster scale changes. Therefore, the present invention can fully and reasonably utilize limited resources and has strong flexibility.

[0035] Preferably, before the local end sends the RPC according to the available tokens, the method further comprises the following steps: setting a minimum token limit for the number of available tokens for each RPC subqueue (i.e., the number of available tokens that can guarantee basic work), and the threshold can be set specifically according to the performance of the local end and the basic requirements of the corresponding target; the minimum number of available tokens for each RPC subqueue is above the minimum token limit. The specific implementation method can be: each RPC subqueue has a configurable minimum available token limit field to ensure fault isolation under abnormal conditions.

[0036] The principle of the above steps is: when different targets in one engine belong to different data pools, if an abnormality (failure or heavy load) occurs in a single data pool and causes RPC flow control, all available tokens may be occupied. At this time, the RPC sent to other data pools is also controlled, which causes other targets to fail to work and expands the scope of the fault. To this end, setting a minimum token limit for the RPC subqueue can constrain the abnormally long and excessive use of available tokens in some data pools due to faults within a certain range (because even if there is an abnormality, other data pools still have available tokens and can work), ensuring fault isolation under abnormal conditions, thereby reducing or even avoiding the impact on other normal data pools.

[0037] Furthermore, the method includes the following steps of updating the number of available tokens of each RPC subqueue according to the total token holding time of the RPC subqueue:

[0038] Step A, see Figure 2 As shown, the total token holding time ∑t of each RPC subqueue k within the update period T (i.e., the time window in the figure, which can be initially set to 5 to 7 seconds according to actual business needs and can be flexibly adjusted later) is calculated. k , the calculation formula is: where t now Represents the current moment, Delegate RPC i The moment to get a valid token and send an RPC, Represents the RPC i The moment when the token is released after the callback, the token holding time of a single RPC within the time window period T can be expressed as It can be concluded that ∑t k It can reflect in real time the target working pressure status of the peer end corresponding to each RPC sub-queue in the update cycle T just passed, so this value can be used as the basis for adaptively and dynamically updating the available tokens.

[0039] Step B: According to the total token holding time ∑t of RPC sub-queue k k , calculate the number of available tokens N that need to be reallocated for the current RPC subqueue k k , the calculation formula is: Where N represents the number of available tokens on this end. min represents the minimum token limit of the RPC subqueue, K represents the total number of all RPC subqueues, ∑ K ∑t k Represents the sum of the overall token holding time of each RPC subqueue.

[0040] The principle and effect of the above steps are: if a consistent threshold is used to constrain the number of RPCs of each engine, it is impossible to adapt to the differences in business loads between different engines and targets. For example, when high-load and low-load targets use the same flow control threshold, the performance of the high-load target cannot be met, which restricts the performance of the storage cluster.

[0041] However, according to the above available token update method, it can be achieved that: the RPC subqueue in a relatively busy or queued waiting state will obtain newly added available tokens, so that RPC can be sent; while the RPC subqueue in a relatively idle state will have the number of available tokens reduced. Therefore, the present invention can dynamically evaluate the working pressure of the peer target through the overall token holding time of the RPC subqueue, and then dynamically update the token allocation weight for each peer target, adaptively adjust the number of available tokens according to the peer pressure state, and make up for the busy by taking advantage of the idle, so as to maximize the system performance based on limited resources.

[0042] At the same time, the above steps also combine the minimum token limit in the previous steps, so that even if the number of available tokens is reduced in the relatively idle RPC subqueue, the minimum available tokens can always be guaranteed to be available, and new RPC sending requests will not be blocked. Even if some targets have a large number of RPC timeouts due to abnormalities, and the overall holding time of available tokens is relatively long, it will not affect the RPC sent to other targets too much, and fault isolation between multiple data pools in the storage cluster can be achieved.

[0043] On this basis, the method also includes the following steps: starting a timed task in the working cycle of this end, and in order to disperse the computing pressure, each RPC automatically calculates the length of time the RPC will hold the available token in the next update cycle T when it is called back, and adds it to the overall token holding time of the RPC sub-queue to which it belongs. In this way, after the update time is reached, the allocation weight can be immediately calculated according to the overall token holding time of each sub-RPC queue and the update of the available token can be completed.

[0044] At the same time, considering that in a distributed storage system, the failure exit of the storage engine, the target exit caused by hard disk damage, the expansion and contraction of the storage node and the expansion and contraction of the hard disk will all trigger the update of the pool map (data pool), resulting in a change in the target set of the opposite end of this client. Therefore, when the present invention monitors the update of the pool map, it is also necessary to trigger a dynamic update of the available tokens. That is, restart this method, that is, according to the update of the pool map, after the corresponding increase or decrease of the RPC subqueue, use the modified update cycle and refresh the cycle starting point.

[0045] Preferably, in this method, for each target of the opposite end for sending and receiving RPC, an RPC subqueue is created on the local end, and the process of allocating available tokens to each RPC subqueue according to the memory of the local end includes: creating an RPC subqueue for each target of the opposite end according to the known information in the pool map, and the subqueue is implemented using a linked list, and a single linked list or a double linked list can be used according to business needs; considering memory saving, the linked list entry of each subqueue only stores a pointer to the original RPC flow control queue entry. The initial number of available tokens of the RPC subqueue is the total number of available tokens available in the local end memory divided by the number of RPC subqueues.

[0046] Preferably, the process of sending RPC by this end according to the available token includes: determining whether there is any available token in the current RPC sub-queue before sending, if so, directly sending the RPC and holding an available token, otherwise putting the current RPC into the RPC flow control queue and waiting; after the RPC callback of the current RPC sub-queue releases the available token and returns it to the current RPC sub-queue, the current RPC is sent.

[0047] See below Figure 3 and Figure 4 As shown, the execution order of the present invention, namely the target granularity dynamic RPC flow control mechanism, is explained through an embodiment.

[0048] S1: Create an RPC subqueue for each peer target based on the known information in the pool map. Go to S2.

[0049] S2: Get the total number of available tokens according to the local resource limit, and evenly distribute the initial available tokens to each RPC subqueue. For example, if the number of available tokens corresponding to the local memory limit is 300 and there are 3 RPC subqueues, the initial available tokens for each subqueue are 100. Set the minimum token limit for the number of available tokens for each RPC subqueue, and go to S3.

[0050] S3: This end sends RPC based on the available token. Before sending, it determines whether there is an available token in the current RPC subqueue. If so, it directly sends the RPC and holds a token. Otherwise, it puts the current RPC into the RPC flow control queue and waits. After the current RPC subqueue has an RPC callback to release the token and return it to the current RPC subqueue, it sends the RPC in the flow control queue in the first-in-first-out order and goes to S4.

[0051] S4: When the update cycle is reached, the number of available tokens of each RPC sub-queue is updated according to the total token holding time of each RPC sub-queue in the cycle. The specific process includes: Step A, see Figure 2 As shown, the total token holding time ∑t of each RPC sub-queue k during the update cycle is calculated k , according to ∑t k Calculate the number of available tokens that need to be reallocated to the RPC subqueue, and go back to S3. The above process is executed as follows: when each RPC is called back, it automatically calculates the length of time that the RPC will hold available tokens in the next update cycle T, and adds it to the total token holding time of the RPC subqueue to which it belongs. When the update cycle is reached, the number of available tokens is recalculated and allocated according to the total token holding time of each RPC subqueue, and the next update cycle begins.

[0052] In addition, if the poolmap update is detected during the cycle, the number of available cards in the RPC subqueue will be updated in advance. The special feature of this situation is that there is an increase or decrease in the total number of RPC subqueues, and the current calculation cycle is shorter than the preset calculation cycle, but the calculation formula described in step A and step B can still be used after adjusting the calculation parameters. After the cards are redistributed, the cycle start is refreshed and go to S3.

[0053] The embodiment of the present invention further provides a storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented. It should be noted that the storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM (Read-Only Memory), a RAM (Random Access Memory), a magnetic disk or an optical disk.

[0054] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that runs on the processor, and the processor implements the above method when executing the computer program.

[0055] The distributed storage RPC flow control system based on adaptive dynamic tokens in the embodiment of the present invention includes an RPC sub-queue creation module, an available token allocation module, an RPC sending module and an available token update module arranged at the local end.

[0056] The RPC subqueue creation module is used to: implement the above process of creating an RPC subqueue for each target that sends and receives RPCs; and set the minimum number of tokens for the number of available tokens for each RPC subqueue;

[0057] The available token allocation module is used to: implement the above process of allocating available tokens to each RPC sub-queue according to the memory of the local end;

[0058] The RPC sending module is used to: implement the above process of sending RPC according to the available token;

[0059] The available token update module is used to implement the above process of updating the number of available tokens of each RPC sub-queue according to the overall token holding time of the RPC sub-queue.

[0060] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware implementation, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer-readable storage medium (or a non-transitory medium) and a communication medium (or a temporary medium).

[0061] As is known to those of ordinary skill in the art, the term computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery medium.

[0062] Exemplarily, the computer-readable storage medium may be an internal storage unit of the electronic device of the aforementioned embodiment, such as a hard disk or memory of the electronic device. The computer-readable storage medium may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device.

[0063] The above are only specific implementations of the embodiments of the present invention, but the protection scope of the embodiments of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the embodiments of the present invention, and these modifications or replacements should be included in the protection scope of the embodiments of the present invention. Therefore, the protection scope of the embodiments of the present invention shall be based on the protection scope of the claims.

Claims

1. A distributed storage RPC flow control method based on adaptive dynamic tokens, characterized in that: The method comprises the following steps: creating an RPC subqueue for each target of a work thread of a peer end that sends or receives an RPC on the local end, and allocating available tokens to each RPC subqueue according to the memory of the local end; the process of creating an RPC subqueue for each target of a work thread of a peer end that sends or receives an RPC on the local end, and allocating available tokens to each RPC subqueue according to the memory of the local end comprises: creating an RPC subqueue for each target of the peer end, and the initial available token quantity of the RPC subqueue is the total available token quantity that can be obtained from the memory of the local end, divided by the number of RPC subqueues; after the local end sends an RPC according to the available tokens, updating the available token quantity of each RPC subqueue according to the overall token holding time of the RPC subqueue; The process of updating the number of available tokens of each RPC sub-queue according to the total token holding time of each RPC sub-queue includes: calculating the total token holding time ∑t of each RPC sub-queue k within the update period T k , the calculation formula is: where t now Represents the current moment, Delegate RPC i The moment to get a valid token and send an RPC, Represents the RPC i The moment to release the token after the callback.

2. The distributed storage RPC flow control method based on adaptive dynamic tokens as claimed in claim 1, characterized in that: Before sending the RPC according to the available tokens, the method further includes the following steps: setting a minimum token limit for the number of available tokens for each RPC sub-queue, and the minimum number of available tokens for each RPC sub-queue is above the minimum token limit.

3. The distributed storage RPC flow control method based on adaptive dynamic token as claimed in claim 2, characterized in that: The process of updating the number of available tokens of each RPC sub-queue according to the total token holding time of each RPC sub-queue also includes: k Calculate the number of available tokens N that need to be reallocated for the current RPC subqueue k , the calculation formula is: Where N represents the number of available tokens on the local end. min represents the minimum token limit of the RPC subqueue, K represents the total number of all RPC subqueues, ∑ K ∑t k Represents the sum of the overall token holding time of each RPC subqueue.

4. The distributed storage RPC flow control method based on adaptive dynamic tokens as claimed in claim 3 is characterized in that: The method also includes the following steps: when each RPC is called back, the available token holding time of the RPC in the next update cycle T is calculated, and the time is accumulated to the overall token holding time of the RPC sub-queue to which it belongs.

5. The distributed storage RPC flow control method based on adaptive dynamic tokens as claimed in claim 4 is characterized in that: The method also includes the following steps: when a pool map update is detected, the update cycle is shortened and the cycle starting point is refreshed according to the increase or decrease of the RPC subqueue corresponding to the update of the data pool topology structure pool map, and the subsequent update cycle is restored to the value before the pool map update.

6. The distributed storage RPC flow control method based on adaptive dynamic tokens according to any one of claims 1 to 5, characterized in that: The process of sending RPC according to the available token at this end includes: judging whether there is an available token in the current RPC subqueue before sending, if so, sending RPC and holding an available token, otherwise waiting; the RPC callback of the current RPC subqueue will release the available token and return it to the current RPC subqueue.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program running on the processor, wherein: When a processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A distributed storage RPC flow control system based on adaptive dynamic tokens, characterized in that: The system includes an RPC sub-queue creation module, an available token allocation module, an RPC sending module and an available token update module arranged at the local end; The RPC subqueue creation module is used to: implement the process of creating an RPC subqueue for each work thread target of the other end that sends and receives RPCs at the local end as described in any one of claims 1 to 6; and the process of setting a minimum token limit for the number of available tokens for each RPC subqueue as described in claim 2; The available token allocation module is used to: implement the process of allocating available tokens to each RPC subqueue according to the memory of the local end as described in any one of claims 1 to 6; The RPC sending module is used to: implement the process of sending RPC according to the available token as described in any one of claims 1 to 6; The available token update module is used to: implement the process of updating the number of available tokens of each RPC sub-queue according to the overall token holding time of each RPC sub-queue as described in any one of claims 1 to 6.

Images