File system data flow control method and device based on token bucket, equipment and medium

Abstract

The scheme of the embodiment of the invention comprises the following steps: declaring a token bucket algorithm, wherein the token bucket algorithm comprises declaring a token bucket structure, declaring a token obtaining interface and declaring a token adding interface; defining a flow control logic, wherein the flow control logic comprises a combination of at least two token bucket structures, definition of a token bucket adding interface, definition of a token obtaining failure callback function, addition of a token obtaining from a cache and definition of a token bucket clearing interface; and IOPS flow control and throughput flow control logics are added in read-write operation. According to the scheme provided by the embodiment of the invention, flow limitation is performed on data operations such as file system reading and writing at the client through the token bucket algorithm, so that the use pressure of the client is reduced, and the influence of resource scrambling and resource occupation on the use performance and user experience of the client is prevented; the use pressure of the client is reduced, and the problem that the service life of the server and the disk is shortened due to long-time over-high pressure load can be avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a token bucket-based file system data flow control method, device, equipment and medium. Background Art

[0002] Currently, cloud storage often involves multiple clients using the server resources of the same storage cluster, and the server resources of such a storage cluster are limited.

[0003] When the usage pressure of a client is too high, it will have a negative impact on the usage of other clients. The resulting resource competition and resource occupation will affect the client's performance and user experience. Long-term excessive pressure load will shorten the service life of the server and disk, thereby increasing the probability of problems in the storage cluster in the later stage.

[0004] Therefore, a method is needed to control and manage the transmission traffic in the network, so as to reduce problems such as network congestion caused by network fluctuations. At the same time, while reducing network pressure, the server pressure can be initially controlled, thereby extending the server's working life. Summary of the invention

[0005] The embodiment of the present invention provides a file system data flow control method, device, equipment and medium based on a token bucket to realize the control and management of transmission flow in a network.

[0006] According to one aspect of the present invention, a token bucket-based file system data flow control method is provided, comprising:

[0007] Declare a token bucket algorithm, the token bucket algorithm comprising: declaring a token bucket structure, declaring a token acquisition interface, and declaring an add token interface;

[0008] Define flow control logic, the flow control logic includes: a combination of at least two token bucket structures, define an interface for adding a token bucket, define a callback function for failing to obtain a token, add a method for obtaining a token from a cache, and define an interface for clearing a token bucket;

[0009] Add IOPS flow control and throughput flow control logic for read and write operations.

[0010] Optionally, the token bucket structure is declared, including: adding the total number of tokens, token locks, the number of available tokens, adding token cycles, token bucket capacity and waiting token queue;

[0011] The statement acquires the token interface, including: calling the token acquisition function of the token bucket algorithm, the client requests to acquire the token, if the token acquisition is successful, executing the subsequent business logic; otherwise, executing the logic of the token bucket algorithm failing to acquire the token;

[0012] The declaring adding a token interface includes: creating a thread to periodically add the number of available tokens to a token bucket, wherein the adding number of available tokens and the time interval are defined in the attributes of the token bucket.

[0013] Optionally, the combination of the at least two token bucket structures comprises: declaring a data structure including a flow control lock, a token bucket queue, a token bucket identifier and a token bucket cache token;

[0014] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket or obtaining a token from the token bucket;

[0015] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is searched through the token bucket identifier;

[0016] The token bucket identifier is an index of a token bucket queue;

[0017] The token bucket cache tokens are tokens cached in each token bucket, and the cached tokens are the number of cached tokens corresponding to the last unsuccessful operation.

[0018] Optionally, the definition of adding a token bucket interface includes: when the client is mounted, according to the token bucket identifier and the configuration item, generating a corresponding token bucket and adding it to the token bucket queue;

[0019] The definition of clearing the token bucket interface includes: when the client is uninstalled, gradually clearing the existing token bucket and threads, reading and writing the IOPS token bucket and the throughput token bucket, and executing the addition of the token corresponding thread.

[0020] Optionally, the definition of a callback function for failure to obtain a token includes:

[0021] When the token is not fully acquired or fails, the callback function is called to execute subsequent business logic when the token is added next time;

[0022] In the callback function, the number of tokens requested to be obtained is added to the cache, and the tokens are obtained from the cache when waiting for the next execution.

[0023] Optionally, the adding of obtaining tokens from the cache includes: when the number of tokens is less than the tokens cached in the token bucket, reducing the tokens cached in the token bucket to continue execution.

[0024] Optionally, the read and write operations add IOPS flow control and throughput flow control logic, including:

[0025] Each operation attempts to obtain 1 token from IOPS. If the acquisition fails, wait for a token adding cycle and then try to obtain the token again;

[0026] Each operation attempts to obtain a token for the amount of data to be read or written from IOPS. If the acquisition fails, wait for a token adding cycle before trying again to obtain the token.

[0027] According to another aspect of the present invention, a file system data flow control device based on a token bucket is provided, comprising:

[0028] A declaration unit, used for declaring a token bucket algorithm, wherein the token bucket algorithm includes: declaring a token bucket structure, declaring a token acquisition interface, and declaring an add token interface;

[0029] A definition unit, used to define a flow control logic, wherein the flow control logic includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failing to obtain a token, adding an interface for obtaining a token from a cache, and defining an interface for clearing a token bucket;

[0030] Add a unit to add IOPS flow control and throughput flow control logic for read and write operations.

[0031] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0032] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the token bucket-based file system data flow control method described in any embodiment of the present invention.

[0033] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the token bucket-based file system data flow control described in any embodiment of the present invention when executed.

[0034] The scheme of the embodiment of the present invention includes: declaring a token bucket algorithm, the token bucket algorithm includes: declaring a token bucket structure, declaring a token acquisition interface and declaring a token addition interface; defining flow control logic, the flow control logic includes: a combination of at least two token bucket structures, defining an add token bucket interface, defining a token acquisition failure callback function, adding a token acquisition from a cache and defining a token bucket clearing interface; adding IOPS flow control and throughput flow control logic for read and write operations. The scheme of the embodiment of the present invention uses a token bucket algorithm on the client to perform flow restrictions on data operations such as file system reading and writing, reduce the client's usage pressure, prevent resource contention and resource occupation from affecting the client's usage performance and user experience, and in addition, reducing the client's usage pressure can avoid the problem of being in an excessively high pressure load for a long time, which shortens the service life of the server and disk.

[0035] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 It is a flow chart of a file system data flow control method based on a token bucket provided by an embodiment of the present invention;

[0038] Figure 2 It is a logical schematic diagram of a file system data flow control method based on a token bucket provided by an embodiment of the present invention;

[0039] Figure 3 It is a structural schematic diagram of a file system data flow control device based on a token bucket provided by an embodiment of the present invention;

[0040] Figure 4 The present invention is a schematic diagram of the structure of an electronic device for implementing the token bucket-based file system data flow control according to an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0042] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a file system data flow control method based on a token bucket, and the method may include the following steps:

[0044] S110, declaring a token bucket algorithm, the token bucket algorithm comprising: declaring a token bucket structure, declaring a token acquisition interface, and declaring a token addition interface.

[0045] The token bucket is a technical mechanism for traffic shaping and flow control. The token bucket puts tokens into the bucket at a certain rate. For example, 10 tokens are put into the bucket per second. When a data message needs to be sent, a token is taken out of the bucket. If there are enough tokens in the bucket, the message can be sent; if there are no tokens in the bucket, the message needs to wait until there are enough tokens.

[0046] Token buckets can effectively absorb burst traffic in the network. Without flow control, data may be sent in bursts, which may cause network congestion and performance degradation. By storing tokens and issuing tokens at a fixed rate, token buckets can ensure that the data transmission rate does not exceed the set limit, thereby avoiding the impact of burst traffic on the network.

[0047] In an embodiment of the present invention, the token bucket structure is declared, including: adding the total number of tokens, token locks, the number of available tokens, adding token cycles, token bucket capacity, and waiting token queues;

[0048] Specifically, the total number of added tokens is the total number of tokens added per second;

[0049] The token lock is an exclusive lock for operating the attributes in the token bucket. It is locked when modifying the token bucket attributes, obtaining tokens, and adding tokens.

[0050] The token adding cycle is the time interval between two token adding cycles. The number of tokens added each time is the number of tokens that should be added in this time period, that is, the number of tokens that should be added at the current time minus the number of tokens that should be added in the last token adding cycle.

[0051] The number of available tokens is the number of tokens in the token bucket that can be used for customer operations, which generally does not exceed the token bucket capacity.

[0052] The token bucket capacity is the maximum number of tokens in the current token bucket, which is generally the upper limit of tokens that the client can obtain.

[0053] Waiting for token queue: When the acquisition of token fails, the corresponding operation will be added to the waiting token queue.

[0054] The statement acquires the token interface, including: calling the token acquisition function of the token bucket algorithm, the client requests to acquire the token, if the token acquisition is successful, executing the subsequent business logic; otherwise, executing the logic of the token bucket algorithm failing to acquire the token;

[0055] In the token acquisition interface, when the waiting token queue is not empty, the token acquisition fails, otherwise it attempts to obtain the number of tokens from the available tokens. If the token is obtained, the available token number is updated. If the number of tokens is insufficient or the token acquisition fails, the token to be obtained and the operation are added to the waiting token queue, waiting for the next token addition to be executed.

[0056] For example, for a token bucket that limits the read operation IOPS to 1000, 1300 read operations are sent in the current token time period, of which the last 300 read operations will wait for the next token time period to retry to obtain the token.

[0057] The declaring adding a token interface includes: creating a thread to periodically add the number of available tokens to a token bucket, wherein the adding number of available tokens and the time interval are defined in the attributes of the token bucket.

[0058] In a fixed thread, the token adding interface is called every token adding cycle. The difference of the current token time cycle is added as the number of tokens added to the number of available tokens, and the token acquisition requests in the waiting token queue are executed. Finally, the request for successful token acquisition is removed from the waiting token queue, and the callback function of the successful token acquisition request is called.

[0059] For example, for a token bucket that limits read operation IOPS to 1000, 1300 read operations are sent within the current token time period, of which the last 300 read operations are added to the waiting token queue.

[0060] S120, define flow control logic, the flow control logic includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failing to obtain a token, adding obtaining a token from a cache, and defining an interface for clearing a token bucket.

[0061] Flow control is a solution to limit the communication traffic between the client and the server. In a broad sense, it can be understood as limiting the request frequency and content of the client. The use of flow control can control and manage the transmission traffic in the network, thereby reducing network congestion and other problems caused by network fluctuations. At the same time, while reducing network pressure, it also allows the server pressure to be initially controlled, thereby extending the working life of the server.

[0062] In an embodiment of the present invention, a combination of at least two token bucket structures includes: declaring a data structure including a flow control lock, a token bucket queue, a token bucket identifier, and a token bucket cache token;

[0063] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket or obtaining a token from the token bucket;

[0064] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is searched through the token bucket identifier;

[0065] The token bucket identifier is an index of a token bucket queue;

[0066] The token bucket cache tokens are tokens cached in each token bucket, and the cached tokens are the number of cached tokens corresponding to the last unsuccessful operation.

[0067] In the embodiment of the present invention, a token bucket adding interface is defined, including: when the client is mounted, according to the token bucket identifier and the configuration item, a corresponding token bucket is generated and added to the token bucket queue.

[0068] Specifically, according to the five values ​​of read IOPS, write IOPS, read throughput, write throughput, and timing period in the configuration, in the add token bucket interface, corresponding token buckets are created in sequence and added to the token bucket queue.

[0069] In the embodiment of the present invention, a token bucket clearing interface is defined, including: when the client is uninstalled, gradually clearing the existing token buckets and threads, reading and writing the IOPS token bucket and the throughput token bucket, and executing the thread corresponding to the addition of tokens.

[0070] In an embodiment of the present invention, a callback function for failure to obtain a token is defined, including:

[0071] When the token is not fully acquired or fails, the callback function is called to execute subsequent business logic when the token is added next time;

[0072] In the callback function, the number of tokens requested to be obtained is added to the cache, and the tokens are obtained from the cache when waiting for the next execution.

[0073] Specifically, in the callback function for failing to obtain a token, the number of tokens requested to be obtained is added to the cache, and the token can be directly obtained from the cache when waiting for the next execution.

[0074] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations are sent in the current token time period, of which the last 300 read operations fail to obtain tokens, and 300 token bucket cache tokens will be generated when waiting for the next token to be added.

[0075] In the embodiment of the present invention, obtaining tokens from the cache is added, including: when the number of tokens is less than the tokens cached in the token bucket, reducing the tokens cached in the token bucket to continue execution.

[0076] For example, for a token bucket with a read operation IOPS limit of 1000, 1300 read operations are sent within the current token time period, of which the last 300 read operations fail to obtain tokens. 300 token bucket cache tokens are generated while waiting for the next token time period, and the number of tokens that can be obtained in the next period is 300 token bucket cache tokens and 700 available tokens.

[0077] S130, add IOPS flow control and throughput flow control logic for read and write operations.

[0078] In the embodiment of the present invention, the read and write operations add IOPS flow control and throughput flow control logic, including:

[0079] Each operation attempts to obtain 1 token from IOPS. If the acquisition fails, wait for a token adding cycle and then try to obtain the token again;

[0080] Each operation attempts to obtain a token for the amount of data to be read or written from IOPS. If the acquisition fails, wait for a token adding cycle before trying again to obtain the token.

[0081] Figure 2 This is a logical diagram of a token bucket-based file system data flow control method provided by an embodiment of the present invention. This solution uses a token bucket algorithm on the client to implement flow restrictions on file system read and write and other data operations. The specific restrictions are the IOPS and throughput of read and write operations. When a large number of read and write requests are accessed, the read and write operations need to perform the token acquisition operation under the exclusive lock of the flow control lock. If the acquisition is successful, the read and write business logic will continue to be executed. Otherwise, the token to be acquired will be converted into a token bucket cache token and wait for the next token addition cycle to be executed.

[0082] The flow control algorithm used in this solution is the token bucket algorithm. First, the token and token bucket are defined. The client requests to execute the request processing logic after obtaining the token, otherwise it waits to obtain the token. The token bucket is a structure for storing tokens. The scheduled task regularly puts a fixed number of tokens into the token bucket, and the client obtains the token by accessing the token bucket. This solution ensures that the client performs no more restrictive operations than the number of tokens put in each regular time period in the long run. This limits the usage pressure of the client.

[0083] like Figure 3 As shown, an embodiment of the present invention provides a file system data flow control device based on a token bucket, and the device may include the following units:

[0084] A declaration unit 310 is used to declare a token bucket algorithm, wherein the token bucket algorithm includes: declaring a token bucket structure, declaring a token acquisition interface, and declaring an add token interface;

[0085] A definition unit 320 is used to define a flow control logic, wherein the flow control logic includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failing to obtain a token, adding an interface for obtaining a token from a cache, and defining an interface for clearing a token bucket;

[0086] The adding unit 330 is used to add IOPS flow control and throughput flow control logic for read and write operations.

[0087] Optionally, the declaration unit 310 is specifically used to: add the total number of tokens, token locks, the number of available tokens, the period for adding tokens, the capacity of the token bucket, and the waiting token queue;

[0088] The statement acquires the token interface, including: calling the token acquisition function of the token bucket algorithm, the client requests to acquire the token, if the token acquisition is successful, executing the subsequent business logic; otherwise, executing the logic of the token bucket algorithm failing to acquire the token;

[0089] The declaring adding a token interface includes: creating a thread to periodically add the number of available tokens to a token bucket, wherein the adding number of available tokens and the time interval are defined in the attributes of the token bucket.

[0090] Optionally, the definition unit 320 is specifically used to: declare a data structure including a flow control lock, a token bucket queue, a token bucket identifier and a token bucket cache token;

[0091] The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket or obtaining a token from the token bucket;

[0092] The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is searched through the token bucket identifier;

[0093] The token bucket identifier is an index of a token bucket queue;

[0094] The token bucket cache tokens are tokens cached in each token bucket, and the cached tokens are the number of cached tokens corresponding to the last unsuccessful operation.

[0095] Optionally, definition unit 320 is specifically used to: when the client is mounted, generate a corresponding token bucket and add it to the token bucket queue according to the token bucket identifier and configuration items; when the client is unmounted, gradually clear the existing token bucket and threads, read and write the IOPS token bucket and throughput token bucket, and execute the addition of the token corresponding thread.

[0096] Optionally, definition unit 320 is specifically used to: when the token acquisition is incomplete or fails, the callback function is called to execute subsequent business logic when the token is added next time; in the callback function, the number of tokens requested is added to the cache, and the token is obtained from the cache when waiting for the next execution.

[0097] The definition unit 320 is specifically used to: when the number of tokens is less than the tokens cached in the token bucket, reduce the tokens cached in the token bucket to continue execution.

[0098] Optionally, the adding unit 330 is specifically used to: attempt to obtain 1 token from IOPS for each operation, if the acquisition fails, wait for a token adding cycle and then try to obtain the token again; attempt to obtain a token for the amount of read and write data from IOPS for each operation, if the acquisition fails, wait for a token adding cycle and then try to obtain the token again.

[0099] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the file system data flow control device based on the token bucket. In other embodiments of the present invention, the file system data flow control based on the token bucket may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0100] The information interaction, execution process and other contents between the units in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0101] Figure 4A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0102] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0103] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as file system data flow control based on token buckets.

[0105] In some embodiments, the token bucket-based file system data flow control may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the token bucket-based file system data flow control described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform token bucket-based file system data flow control in any other appropriate manner (e.g., by means of firmware).

[0106] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0107] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0108] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0109] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0110] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0111] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A file system data flow control method based on a token bucket, characterized in that: include: Declare a token bucket algorithm, the token bucket algorithm comprising: declaring a token bucket structure, declaring a token acquisition interface, and declaring an add token interface; Define flow control logic, the flow control logic includes: a combination of at least two token bucket structures, define an interface for adding a token bucket, define a callback function for failing to obtain a token, add a method for obtaining a token from a cache, and define an interface for clearing a token bucket; Add IOPS flow control and throughput flow control logic for read and write operations.

2. The method according to claim 1, characterized in that: The token bucket structure is declared, including: the total number of added tokens, token locks, the number of available tokens, the period for adding tokens, the capacity of the token bucket and the waiting token queue; The statement acquires the token interface, including: calling the token acquisition function of the token bucket algorithm, the client requests to acquire the token, if the token acquisition is successful, executing the subsequent business logic; otherwise, executing the logic of the token bucket algorithm failing to acquire the token; The declaring adding a token interface includes: creating a thread to periodically add the number of available tokens to a token bucket, wherein the adding number of available tokens and the time interval are defined in the attributes of the token bucket.

3. The method according to claim 1, characterized in that: The combination of at least two token bucket structures includes: declaring a data structure including a flow control lock, a token bucket queue, a token bucket identifier and a token bucket cache token; The flow control lock is an exclusive lock for token bucket operations, which is locked when adding a token bucket or obtaining a token from the token bucket; The token bucket queue is a list of at least two token bucket structures, and the corresponding token bucket is searched through the token bucket identifier; The token bucket identifier is an index of a token bucket queue; The token bucket cache tokens are tokens cached in each token bucket, and the cached tokens are the number of cached tokens corresponding to the last unsuccessful operation.

4. The method according to claim 1, characterized in that The definition of adding a token bucket interface includes: when the client is mounted, according to the token bucket identifier and the configuration item, generating a corresponding token bucket and adding it to the token bucket queue; The definition of clearing the token bucket interface includes: when the client is uninstalled, gradually clearing the existing token bucket and threads, reading and writing the IOPS token bucket and the throughput token bucket, and executing the addition of the token corresponding thread.

5. The method according to claim 1, characterized in that The definition of the token acquisition failure callback function includes: When the token is not fully acquired or fails, the callback function is called to execute subsequent business logic when the token is added next time; In the callback function, the number of tokens requested to be obtained is added to the cache, and the tokens are obtained from the cache when waiting for the next execution.

6. The method according to claim 1, characterized in that The adding of obtaining tokens from the cache includes: when the number of tokens is less than the tokens cached in the token bucket, reducing the tokens cached in the token bucket to continue execution.

7. The method according to claim 1, characterized in that The read and write operations add IOPS flow control and throughput flow control logic, including: Each operation attempts to obtain 1 token from IOPS. If the acquisition fails, wait for a token adding cycle and then try to obtain the token again; Each operation attempts to obtain a token for the amount of data to be read or written from IOPS. If the acquisition fails, wait for a token adding cycle before trying again to obtain the token.

8. A file system data flow control device based on a token bucket, characterized in that: include: A declaration unit, used for declaring a token bucket algorithm, wherein the token bucket algorithm includes: declaring a token bucket structure, declaring a token acquisition interface, and declaring an add token interface; A definition unit, used to define a flow control logic, wherein the flow control logic includes: a combination of at least two token bucket structures, defining an interface for adding a token bucket, defining a callback function for failing to obtain a token, adding an interface for obtaining a token from a cache, and defining an interface for clearing a token bucket; Add a unit to add IOPS flow control and throughput flow control logic for read and write operations.

9. An electronic device, characterized in that include: at least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the token bucket-based file system data flow control method described in any one of claims 1-7.

10. A computer readable medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the token bucket-based file system data flow control method described in any one of claims 1 to 7 when executed.

Images