Methods, apparatus, devices, and media for data flow control of file systems based on token buckets.

Abstract

The solution of this invention includes: declaring a token bucket algorithm, which includes: declaring a token bucket structure, declaring a token acquisition interface, and declaring a token addition interface; defining flow control logic, which includes: a combination of at least two token bucket structures, defining a token addition interface, defining a callback function for token acquisition failure, adding a token acquisition from the cache, and defining a token clearing interface; and adding IOPS flow control and throughput flow control logic for read and write operations. The solution of this invention uses the token bucket algorithm on the client side to implement traffic limits on data operations such as file system read and write operations, reducing the pressure on the client and preventing resource contention and resource consumption from affecting client performance and user experience. Furthermore, reducing the pressure on the client can avoid the problem of prolonged high load reducing the lifespan of servers and disks.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for data flow control of a file system based on a token bucket. Background Technology

[0002] Currently, cloud storage often involves multiple clients using the server resources of the same storage cluster, and the server resources of such storage clusters are limited.

[0003] When one client experiences high usage pressure, it negatively impacts other clients, leading to resource contention and excessive resource consumption that affects client performance and user experience. Prolonged periods of excessive load can shorten the lifespan of servers and disks, increasing the likelihood of storage cluster problems later on.

[0004] Therefore, a way is needed to control and manage the traffic transmitted in the network, thereby reducing network congestion and other problems caused by network fluctuations. At the same time, while reducing network pressure, it can also initially control server pressure, thereby extending the service life of the server. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and medium for controlling file system data flow based on token bucket, so as to control and manage the transmission traffic in the network.

[0006] According to one aspect of the present invention, a file system data flow control method based on token bucket is provided, comprising:

[0007] The token bucket algorithm is declared, which includes: declaring the token bucket structure, declaring the token acquisition interface, and declaring the token addition interface;

[0008] Define flow control logic, which includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failure to acquire a token, adding an interface for acquiring a token from the cache, and defining an interface for clearing the token bucket;

[0009] Add IOPS flow control and throughput flow control logic to read and write operations.

[0010] Optionally, the token bucket structure includes: total number of tokens to be added, token lock, number of available tokens, token addition cycle, token bucket capacity, and waiting token queue;

[0011] The declared token acquisition interface includes: calling the token bucket algorithm's token acquisition function, the client requests to acquire a token, and if the token acquisition is successful, the subsequent business logic is executed; otherwise, the logic for the token bucket algorithm failing to acquire a token is executed.

[0012] The statement adds a token interface, including: creating a thread to periodically add the number of available tokens to the token bucket, wherein the number of available tokens added and the time interval are defined in the properties of the token bucket.

[0013] Optionally, the combination of the at least two token bucket structures includes: declaring a data structure containing a flow control lock, a token bucket queue, a token bucket identifier, and a token bucket cache token;

[0014] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket and obtaining a token from the token bucket.

[0015] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is found by the token bucket identifier;

[0016] The token bucket identifier is the index of the token bucket queue;

[0017] The token bucket cache tokens are the tokens cached in each token bucket, and the number of cached tokens corresponds to the number of cached tokens for the last operation that failed to execute.

[0018] Optionally, the definition of the token bucket interface includes: when the client mounts the token bucket, generating the corresponding token bucket and adding it to the token bucket queue according to the token bucket identifier and configuration items;

[0019] The defined token bucket clearing interface includes: gradually clearing existing token buckets and threads when the client uninstalls, reading and writing IOPS token buckets and throughput token buckets, and executing the thread corresponding to the added token.

[0020] Optionally, the definition of the token acquisition failure callback function includes:

[0021] If the token acquisition is incomplete or fails, the callback function will be called to execute the subsequent business logic when the token is added again.

[0022] In the callback function, the number of tokens requested is added to the cache, and the tokens are retrieved from the cache the next time the function is executed.

[0023] Optionally, the addition of obtaining tokens from the cache includes: reducing the number of tokens in the token bucket cache to continue execution when the number of tokens is less than the number of tokens in the token bucket cache.

[0024] Optionally, the read / write operations are supplemented with IOPS flow control and throughput flow control logic, including:

[0025] Each operation attempts to acquire 1 token from IOPS. If the acquisition fails, wait for one token acquisition cycle and then try to acquire the token again.

[0026] Each operation attempts to obtain a token for the amount of data read and written from IOPS. If the attempt fails, it waits for one token addition cycle before attempting to obtain the token again.

[0027] According to another aspect of the present invention, a token bucket-based file system data flow control device is provided, comprising:

[0028] A declaration unit is used to declare the token bucket algorithm, which includes: declaring the token bucket structure, declaring the token acquisition interface, and declaring the token addition interface;

[0029] A definition unit is used to define flow control logic, which includes: a combination of at least two token bucket structures, a definition of an interface for adding a token bucket, a definition of a callback function for failure to acquire a token, an interface for adding a token to be acquired from the cache, and a definition of an interface for clearing the token bucket.

[0030] Add a unit to add IOPS flow control and throughput flow control logic for read and write operations.

[0031] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0032] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the token bucket-based file system data flow control method according to any embodiment of the present invention.

[0033] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the token bucket-based file system data flow control described in any embodiment of the present invention.

[0034] The solution of this invention includes: declaring a token bucket algorithm, which includes: declaring a token bucket structure, declaring a token acquisition interface, and declaring a token addition interface; defining flow control logic, which includes: a combination of at least two token bucket structures, defining a token addition interface, defining a callback function for token acquisition failure, adding a token acquisition from the cache, and defining a token clearing interface; and adding IOPS flow control and throughput flow control logic for read and write operations. The solution of this invention uses the token bucket algorithm on the client side to implement traffic limits on data operations such as file system read and write operations, reducing the pressure on the client and preventing resource contention and resource consumption from affecting client performance and user experience. Furthermore, reducing the pressure on the client can avoid the problem of prolonged high load reducing the lifespan of servers and disks.

[0035] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart of a file system data flow control method based on a token bucket according to an embodiment of the present invention;

[0038] Figure 2 This is a logical schematic diagram of a file system data flow control method based on a token bucket according to an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of a data flow control device for a file system based on a token bucket, provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the structure of an electronic device that implements token bucket-based file system data flow control according to an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0043] like Figure 1 As shown, this embodiment of the invention provides a file system data flow control method based on a token bucket, which may include the following steps:

[0044] S110, Declare the token bucket algorithm, which includes: declaring the token bucket structure, declaring the token acquisition interface, and declaring the token addition interface.

[0045] A token bucket is a traffic shaping and flow control mechanism. Tokens are added to the bucket at a set rate, for example, 10 tokens per second. When a data packet needs to be sent, a token is removed from the bucket. If there are enough tokens in the bucket, the packet can be sent; if there are no tokens, the packet must wait until enough are available.

[0046] Token buckets effectively absorb bursts of traffic in a network. Without flow control, data can be sent in bursts, potentially causing network congestion and performance degradation. Token buckets, by storing tokens and issuing them at a fixed rate, ensure that the data transmission rate does not exceed a set limit, thus preventing bursts of traffic from impacting the network.

[0047] In this embodiment of the invention, the token bucket structure includes: total number of tokens to be added, token lock, number of available tokens, token addition period, token bucket capacity, and waiting token queue;

[0048] Specifically, the total number of tokens added is the total number of tokens added per second;

[0049] A token lock is an exclusive lock for manipulating attributes in a token bucket. It is used when modifying token bucket attributes, acquiring tokens, or adding tokens.

[0050] The token addition cycle is the time interval between two token additions. The number of tokens added each time is the number of tokens that should have been added during this time period, which is the number of tokens that should have been added at the current time minus the number of tokens that should have been added in the previous token addition cycle.

[0051] The number of available tokens is the number of tokens in the token bucket that can be used for customer operations, and it generally does not exceed the token bucket capacity.

[0052] The token bucket capacity is the maximum number of tokens currently in the token bucket, which is generally the maximum number of tokens that a client can acquire.

[0053] The waiting token queue is where the corresponding operation is added when acquiring a token fails.

[0054] The declared token acquisition interface includes: calling the token bucket algorithm's token acquisition function, the client requests to acquire a token, and if the token acquisition is successful, the subsequent business logic is executed; otherwise, the logic for the token bucket algorithm failing to acquire a token is executed.

[0055] In the token acquisition interface, if the waiting token queue is not empty, acquiring a token fails; otherwise, it attempts to acquire tokens from the available token count. If a token is acquired, the available token count is updated. If the number of tokens acquired is insufficient or acquiring a token fails, the tokens to be acquired and the operation are added to the waiting token queue, awaiting execution the next time a token is added.

[0056] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations were sent within the current token time period, of which the last 300 read operations will wait for the next token time period to retry acquiring a token.

[0057] The statement adds a token interface, including: creating a thread to periodically add the number of available tokens to the token bucket, wherein the number of available tokens added and the time interval are defined in the properties of the token bucket.

[0058] In a fixed thread, the add token interface is called every token addition cycle. The difference between the current token time cycle is added as the number of tokens to be added to the available token count. Requests to acquire tokens in the waiting token queue are executed. Finally, requests that successfully acquire tokens are removed from the waiting token queue, and the callback function for successful token acquisition requests is called.

[0059] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations were sent within the current token time period, and the last 300 read operations were added to the waiting token queue.

[0060] S120. Define flow control logic, which includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failure to obtain a token, adding an interface for obtaining a token from the cache, and defining an interface for clearing the token bucket.

[0061] Flow control, or traffic management, is a scheme to limit the communication traffic between clients and servers. Broadly speaking, it can be understood as limiting the frequency and content of client requests. Flow control enables the control and management of traffic transmitted in the network, thereby reducing network congestion caused by network fluctuations. Simultaneously, while reducing network pressure, it also allows for initial control of server load, thus extending the server's lifespan.

[0062] In an embodiment of the invention, the combination of at least two token bucket structures includes: declaring a data structure containing a flow control lock, a token bucket queue, a token bucket identifier, and a token bucket cache token;

[0063] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket and obtaining a token from the token bucket.

[0064] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is found by the token bucket identifier;

[0065] The token bucket identifier is the index of the token bucket queue;

[0066] The token bucket cache tokens are the tokens cached in each token bucket, and the number of cached tokens corresponds to the number of cached tokens for the last operation that failed to execute.

[0067] In this embodiment of the invention, an interface for adding a token bucket is defined, including: when the client mounts the token bucket, generating a corresponding token bucket based on the token bucket identifier and configuration items and adding it to the token bucket queue.

[0068] Specifically, based on the five values ​​of read IOPS, write IOPS, read throughput, write throughput, and timing period in the configuration, the corresponding token buckets are created sequentially in the add token bucket interface and added to the token bucket queue.

[0069] In this embodiment of the invention, a token bucket clearing interface is defined, including: when the client uninstalls, gradually clearing the existing token bucket and threads, reading and writing the IOPS token bucket and the throughput token bucket, and executing the addition of the corresponding thread for the token.

[0070] In this embodiment of the invention, a callback function for failure to obtain a token is defined, including:

[0071] If the token acquisition is incomplete or fails, the callback function will be called to execute the subsequent business logic when the token is added again.

[0072] In the callback function, the number of tokens requested is added to the cache, and the tokens are retrieved from the cache the next time the function is executed.

[0073] Specifically, in the callback function for failed token acquisition, the number of tokens requested is added to the cache, so that the tokens can be retrieved directly from the cache the next time the process is executed.

[0074] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations were sent within the current token time period. The last 300 read operations failed to acquire tokens, and 300 tokens will be generated as tokens are cached in the token bucket when the next token is added.

[0075] In this embodiment of the invention, adding the method of obtaining tokens from the cache includes: when the number of tokens is less than the number of tokens cached in the token bucket, reducing the number of tokens cached in the token bucket to continue execution.

[0076] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations were sent in the current token time period. The last 300 read operations failed to acquire tokens and will generate 300 token bucket cache tokens when waiting for the next token time period. The number of tokens that can be acquired in the next period is 300 token bucket cache tokens and 700 available tokens.

[0077] S130: Add IOPS flow control and throughput flow control logic to read and write operations.

[0078] In this embodiment of the invention, IOPS flow control and throughput flow control logic are added to read and write operations, including:

[0079] Each operation attempts to acquire 1 token from IOPS. If the acquisition fails, wait for one token acquisition cycle and then try to acquire the token again.

[0080] Each operation attempts to obtain a token for the amount of data read and written from IOPS. If the attempt fails, it waits for one token addition cycle before attempting to obtain the token again.

[0081] Figure 2 This is a logical diagram illustrating a token bucket-based file system data flow control method according to an embodiment of the present invention. This solution uses a token bucket algorithm on the client side to implement flow restrictions on file system read and write operations, specifically limiting the IOPS and throughput of read and write operations. When a large number of read and write requests are requested, the read and write operations need to perform a token acquisition operation under the exclusive lock of the flow control lock. If the acquisition is successful, the read and write business logic continues; otherwise, the token to be acquired is converted into a token bucket cache token, waiting for the next token addition cycle.

[0082] The flow control algorithm used in this scheme is the token bucket algorithm. First, tokens and the token bucket are defined. After a client obtains a token, its request processing logic is executed; otherwise, it waits to obtain a token. The token bucket is the structure for storing tokens. A scheduled task periodically adds a fixed number of tokens to the token bucket, and clients obtain tokens by accessing the token bucket. This scheme ensures that, in the long run, the restricted operations performed by the client within each periodic time frame do not exceed the number of tokens added, thus limiting the client's usage load.

[0083] like Figure 3 As shown, this embodiment of the invention provides a file system data flow control device based on a token bucket, which may include the following units:

[0084] Declaration unit 310 is used to declare the token bucket algorithm, which includes: declaring the token bucket structure, declaring the token acquisition interface and declaring the token addition interface;

[0085] Definition unit 320 is used to define flow control logic, which includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failure to obtain a token, adding an interface for obtaining a token from the cache, and defining an interface for clearing the token bucket;

[0086] Add unit 330 to add IOPS flow control and throughput flow control logic for read and write operations.

[0087] Optionally, the declaration unit 310 is specifically used for: adding the total number of tokens, the token lock, the number of available tokens, the token addition cycle, the token bucket capacity, and the waiting token queue;

[0088] The declared token acquisition interface includes: calling the token bucket algorithm's token acquisition function, the client requests to acquire a token, and if the token acquisition is successful, the subsequent business logic is executed; otherwise, the logic for the token bucket algorithm failing to acquire a token is executed.

[0089] The statement adds a token interface, including: creating a thread to periodically add the number of available tokens to the token bucket, wherein the number of available tokens added and the time interval are defined in the properties of the token bucket.

[0090] Optionally, define unit 320, specifically for: declaring a data structure containing a flow control lock, a token bucket queue, a token bucket identifier, and a token bucket cache token;

[0091] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket and obtaining a token from the token bucket.

[0092] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is found by the token bucket identifier;

[0093] The token bucket identifier is the index of the token bucket queue;

[0094] The token bucket cache tokens are the tokens cached in each token bucket, and the number of cached tokens corresponds to the number of cached tokens for the last operation that failed to execute.

[0095] Optionally, define unit 320, which is specifically used for: generating the corresponding token bucket and adding it to the token bucket queue according to the token bucket identifier and configuration items when the client is mounted; and gradually clearing the existing token buckets and threads when the client is unmounted, reading and writing the IOPS token bucket and throughput token bucket, and executing the thread corresponding to the added token.

[0096] Optionally, unit 320 is defined as follows: when the token acquisition is incomplete or fails, a callback function is called to execute subsequent business logic when the token is added again; in the callback function, the number of tokens requested is added to the cache, and the tokens are retrieved from the cache when the next execution is performed.

[0097] Define unit 320, specifically used to: reduce the number of tokens in the token bucket cache to continue execution when the number of tokens is less than the number of tokens in the token bucket cache.

[0098] Optionally, add unit 330, specifically for: attempting to obtain 1 token from IOPS for each operation; if the acquisition fails, waiting for one token addition cycle before attempting to obtain the token again; and attempting to obtain tokens for the amount of data read / written from IOPS for each operation; if the acquisition fails, waiting for one token addition cycle before attempting to obtain the token again.

[0099] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the token bucket-based file system data flow control device. In other embodiments of the present invention, the token bucket-based file system data flow control may include more or fewer components than illustrated, or combine some components, split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0100] The information interaction and execution process between the various units in the above-mentioned device are based on the same concept as the method embodiment of the present invention, and the specific details can be found in the description of the method embodiment of the present invention, and will not be repeated here.

[0101] Figure 4A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0102] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0103] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0104] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as token bucket-based file system data flow control.

[0105] In some embodiments, token bucket-based file system flow control can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the token bucket-based file system flow control described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform token bucket-based file system flow control by any other suitable means (e.g., by means of firmware).

[0106] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0108] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0109] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0110] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0111] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0112] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0113] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A token bucket based file system data flow control method, characterized in that, The application comprises: A claim token bucket algorithm, comprising: a claim token bucket structure, a claim token acquisition interface, and a claim token addition interface; Define flow control logic, which includes: combination of at least two token bucket structures, define add token bucket interface, define get token failure callback function, add get token from cache, and define clear token bucket interface; Read-write operation adds IOPS flow control and throughput flow control logic; The claim token bucket structure comprises: total number of added tokens, token lock, available token quantity, token addition period, token bucket capacity, and waiting token queue; The claim token acquisition interface comprises: calling the get token function of the token bucket algorithm, the client requests to acquire tokens, if the token acquisition is successful, the subsequent business logic is executed; otherwise, the token bucket algorithm acquisition failure logic is executed; The claim token addition interface comprises: creating a thread to periodically add available tokens to the token bucket, wherein the number of available tokens and the time interval are defined in the properties of the token bucket; The combination of at least two token bucket structures comprises: declaring a data structure containing a flow control lock, a token bucket queue, a token bucket identifier, and a token bucket cache token; The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket and acquiring a token bucket token; The token bucket queue is a list of at least two token bucket structures, which is found by token bucket identifier; The token bucket identifier is the index of the token bucket queue; The token bucket cache token is the cached token in each token bucket, and the cached token is the number of cached tokens corresponding to the last unsuccessful operation; The define add token bucket interface comprises: when the client is mounted, the corresponding token bucket is generated according to the token bucket identifier and the configuration item, and is added to the token bucket queue; The define clear token bucket interface comprises: when the client is unloaded, the existing token bucket and thread are gradually cleared, the read-write IOPS token bucket and the throughput token bucket are executed, and the add token corresponding thread is executed; The define get token failure callback function comprises: When the token acquisition is not complete or fails, the callback function is called to execute the subsequent business logic when the next token is added; In the callback function, the number of tokens requested to be acquired is added to the cache, and the token is acquired from the cache next time; The add get token from cache comprises: when the number of tokens is less than the token bucket cache token, the token bucket cache token is reduced to continue execution; The read-write operation adds IOPS flow control and throughput flow control logic, comprising: Each operation attempts to acquire 1 token from IOPS, if the acquisition fails, then waits for an add token period before attempting to acquire the token again; Each operation attempts to acquire the read-write data volume token from IOPS, if the acquisition fails, then waits for an add token period before attempting to acquire the token again.

2. An electronic device, characterized by The application comprises: At least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the token bucket-based file system data flow control method in claim 1.

3. A computer readable medium characterized by The computer readable storage medium stores computer instructions for causing a processor to implement the token bucket-based file system data flow control method in claim 1 when executed.

Images