Access control method, device, terminal and storage medium

Abstract

The present application discloses an access control method, device, terminal and storage medium, the method comprising: receiving an access request from a client, then determining a flow restriction method corresponding to the client based on the access request, and then judging whether to allow the client to access a Redis cluster based on the flow restriction method and the access request. The present invention sets an access control policy at the twemproxy proxy middleware layer, that is, identifies the received access request, and selectively intercepts it using a corresponding flow restriction method, thereby avoiding overloading of the entire Redis server cluster and ensuring the stability of the Redis server cluster.

Description

Technical Field

[0001] The present application relates to the technical field of traffic control, and in particular to an access control method, device, terminal and storage medium. Background Art

[0002] The twemproxy proxy middleware is located between the client and the Redis server cluster (i.e., the Redis cluster). It processes the request sent by the client in a certain way (such as sharding) and then forwards it to the real Redis server cluster at the backend. In other words, the client does not directly access the Redis server cluster, but indirectly accesses it through the twemproxy proxy middleware.

[0003] Since the native twemproxy only has basic traffic forwarding capabilities and does not have traffic control capabilities, twemproxy still transparently transmits a large number of requests to the Redis server cluster in the event of unexpected / unexpected traffic scenarios.

[0004] However, in the case of unexpected / unexpected traffic scenarios, twemproxy will pass a large number of requests to the Redis server cluster, causing the entire Redis server cluster to be overloaded due to unexpected traffic, thereby reducing the stability of the Redis server cluster. Summary of the invention

[0005] The main purpose of the present application is to provide an access control method, device, terminal and storage medium to solve the problem of reducing the stability of the Redis server cluster existing in the related art.

[0006] In order to achieve the above objectives, in a first aspect, the present application provides an access control method, comprising:

[0007] Receive access requests from clients;

[0008] Based on the access request, determine the traffic restriction method corresponding to the client;

[0009] Based on the traffic restriction method and access request, determine whether to allow the client to access the Redis cluster.

[0010] In a possible implementation, the flow limiting method is a passive flow limiting method;

[0011] Based on the traffic restriction method and access request, determine whether to allow the client to access the Redis cluster, including:

[0012] The current limiting algorithm corresponding to the passive current limiting method is used to count the client's access traffic;

[0013] Get the traffic IP corresponding to the access traffic;

[0014] Based on the traffic IP and the first IP list corresponding to the whitelist, determine whether the client is allowed to access the Redis cluster.

[0015] In a possible implementation, judging whether to allow the client to access the Redis cluster based on the traffic IP and the first IP list corresponding to the whitelist includes:

[0016] If there is a traffic IP in the first IP list, the client is allowed to access the Redis cluster.

[0017] In a possible implementation, judging whether to allow the client to access the Redis cluster based on the traffic IP and the first IP list corresponding to the whitelist includes:

[0018] If the traffic IP does not exist in the first IP list, determine whether the traffic IP exists in the second IP list corresponding to the blacklist;

[0019] If there is no traffic IP in the second IP list corresponding to the blacklist, and the access traffic does not reach the global current limiting threshold, based on the traffic type corresponding to the traffic IP, determine whether to allow the client to access the Redis cluster.

[0020] In a possible implementation, whether to allow the client to access the Redis cluster is determined based on the traffic type corresponding to the traffic IP, including:

[0021] If the traffic type corresponding to the traffic IP is write traffic, the traffic IP does not reach the write traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster.

[0022] In a possible implementation, whether to allow the client to access the Redis cluster is determined based on the traffic type corresponding to the traffic IP, including:

[0023] If the traffic type corresponding to the traffic IP is read traffic, the traffic IP does not reach the read traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster.

[0024] In a possible implementation, the method further includes:

[0025] Determine whether to enable the collaborative current limiting mode and obtain the determination result;

[0026] Based on the judgment result, determine whether to allow the client to access the Redis cluster.

[0027] In a possible implementation, based on the determination result, determining whether to allow the client to access the Redis cluster includes:

[0028] If the judgment result is no, the client is allowed to access the Redis cluster;

[0029] If the result of the judgment is yes, the current limiting algorithm corresponding to the collaborative current limiting method is used to count the frequency corresponding to the access request;

[0030] Based on the frequency and global current limiting threshold, determine whether to allow the client to access the Redis cluster.

[0031] In a possible implementation, whether to allow a client to access a Redis cluster is determined based on the frequency and the global current limiting threshold, including:

[0032] If the frequency does not reach the global current limit threshold, the client is allowed to access the Redis cluster.

[0033] In a possible implementation, the flow limiting mode is an active flow limiting mode;

[0034] Based on the traffic restriction method and access request, determine whether to allow the client to access the Redis cluster, including:

[0035] The information in the access request is processed using the current limiting algorithm corresponding to the active current limiting method, and the conditional data of the traffic IP corresponding to the access request is screened, where the conditional data at least includes CPU utilization, client QPS, number of slow queries, number of batch requests, and data used to characterize whether to perform master-slave switching;

[0036] Based on the traffic IP condition data and preset judgment rules, determine whether to allow the client to access the Redis cluster.

[0037] In a possible implementation, based on the conditional data of the traffic IP and a preset determination rule, it is determined whether to allow the client to access the Redis cluster, including:

[0038] If the data used to indicate whether the master-slave switch is executed is that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP does not reach the batch request threshold, the client is allowed to access the Redis cluster;

[0039] When the data used to characterize whether the master-slave switch is executed is that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, and the CPU utilization does not meet the preset utilization, the client is allowed to access the Redis cluster.

[0040] In a possible implementation, based on the conditional data of the traffic IP and a preset determination rule, it is determined whether to allow the client to access the Redis cluster, including:

[0041] When the data used to indicate whether to execute the master-slave switch indicates that the master-slave switch is to be executed, the client is not allowed to access the Redis cluster;

[0042] If the data used to indicate whether the master-slave switch is executed indicates that the master-slave switch is not executed and the traffic IP does not reach the write traffic threshold, the client is not allowed to access the Redis cluster;

[0043] If the data used to indicate whether the master-slave switch is executed indicates that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, and the traffic IP does not reach the slow query threshold, the client is not allowed to access the Redis cluster;

[0044] If the data used to characterize whether the master-slave switch is executed is that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP reaches the batch request threshold, the client is not allowed to access the Redis cluster.

[0045] In a second aspect, an embodiment of the present invention provides an access control device, including:

[0046] A receiving module, used for receiving an access request from a client;

[0047] A flow limiting mode determination module is used to determine the flow limiting mode corresponding to the client based on the access request;

[0048] The access control module is used to determine whether to allow the client to access the Redis cluster based on the traffic restriction method and access request.

[0049] In a third aspect, an embodiment of the present invention provides a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above access control methods when executing the computer program.

[0050] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above access control methods are implemented.

[0051] The embodiment of the present invention provides an access control method, device, terminal and storage medium, including: receiving an access request from a client, then determining a flow restriction method corresponding to the client based on the access request, and then judging whether to allow the client to access a Redis cluster based on the flow restriction method and the access request. The present invention sets an access control policy at the twemproxy proxy middleware layer, that is, identifies the received access request, and selectively intercepts it using a corresponding flow restriction method, thereby avoiding overloading of the entire Redis server cluster and ensuring the stability of the Redis server cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0053] Figure 1 is a flow chart of a flow control method provided by an embodiment of the present invention;

[0054] Figure 2 is a system structure diagram of a flow control method provided by an embodiment of the present invention;

[0055] Figure 3 It is a structural schematic diagram of a flow control device of a twemproxy proxy middleware provided in an embodiment of the present invention;

[0056] Figure 4 It is a schematic diagram of configuration information of a passive current limiting method provided by a rule engine configuration module provided in an embodiment of the present invention;

[0057] Figure 5 is a schematic diagram of a thread provided by a traffic statistics aggregation module provided in an embodiment of the present invention;

[0058] Figure 6 is a flow chart for realizing flow control by using a collaborative flow limiting method provided by an embodiment of the present invention;

[0059] Figure 7 is a flow chart for realizing flow control by using active current limiting method provided by an embodiment of the present invention;

[0060] Figure 8 is a schematic structural diagram of a flow control device provided by an embodiment of the present invention;

[0061] Fig. 9 is a schematic diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0064] It should be understood that in various embodiments of the present invention, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0065] It should be understood that in the present invention, "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0066] It should be understood that in the present invention, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0067] It should be understood that in the present invention, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0068] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0069] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0070] First, in order to make the solution of the present invention clear and easy to understand, the professional terms are explained as follows:

[0071] Redis: REmoteDIctionaryServer is a popular key-value storage system in the industry, belonging to the NoSQL category. It supports data structures such as string, hash, set, zset, and list.

[0072] Rediscluster: Redis cluster, but in this patent, it does not refer to the open source redis-cluster mode, but specifically refers to an architecture composed of multiple shardings. Each shard stores a part of the data, and all shards are superimposed to store the full data of the rediscluster.

[0073] twemproxy: The twemproxy project, which is open sourced by Twitter on GitHub, is a proxy middleware for memcache and redis. When users connect to access twemproxy, twemproxy forwards their read and write requests uniformly and distributes them to redis-server or memcache.

[0074] Furthermore, in order to make the objectives, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0075] In one embodiment, Figure 1 As shown, an access control method is provided, comprising the following steps:

[0076] Step S101: receiving an access request from a client;

[0077] Step S102: Based on the access request, determine the traffic restriction method corresponding to the client.

[0078] The twemproxy proxy middleware is located between the client and the Redis server cluster (i.e., the Redis cluster). It processes the request sent by the client (such as sharding) and then forwards it to the real Redis server cluster at the backend. The Redis server cluster includes multiple shardings, each of which stores different data. User requests will be routed to different shards according to the hash value of the key. Each shard will contain at least one master (redis-master) and one slave (redis-slave). For example Figure 2 The Redis server cluster shown in the figure includes three shards, namely shard-1, shard-2 and shard-3. Each of the three shards has a master (redis-master) and a slave (redis-slave).

[0079] Combination Figure 3 , the twemproxy proxy middleware in this application has a flow control device, wherein the flow control device includes a rule engine configuration module, a flow statistics aggregation module, a flow screening and filtering module, and an algorithm plug-in management module. Among them, the rule configuration engine module is mainly for the flow limiting rule configuration. The client configures the corresponding flow limiting rules in twemproxy. After modification, twemproxy can be restarted without restarting it, and hot loading takes effect; the flow statistics aggregation module consumes a small amount of memory to store the access frequency of each IP within a certain period, as well as the TOPIP access records of each type of request; the flow screening and filtering module identifies the unexpected flow, and according to the rule configuration engine, the flow that hits the rule is discarded; the algorithm plug-in management module implements the algorithm of the flow limiting function, which is implemented through a dynamic link library, and is loaded and implemented in twemproxy with a dlopen function call. If the algorithm is upgraded later, the flow limiting function algorithm can be hot-upgraded without restarting twemproxy.

[0080] Through the cooperation of the above four models, the twemproxy proxy middleware identifies the access requests sent by the client and selectively intercepts them using the corresponding traffic limiting method, thereby avoiding overload of the entire redis server cluster and ensuring the stability of the redis server cluster.

[0081] Step S103: Based on the traffic restriction method and the access request, determine whether to allow the client to access the Redis cluster.

[0082] This application provides three flow limiting methods, namely active flow limiting method, passive flow limiting method and coordinated flow limiting method.

[0083] For the passive current limiting method, the rule engine configuration module needs to configure the following information. One part is the configuration parsing, which will be composed in YAML format, and the other part is the built-in engine layer implementation.

[0084] For configuration parsing, Figure 4 As shown, the entire current limiting configuration is in YAML format. The file is divided into three parts:

[0085] Global: This refers to global configuration. The configuration in this area will take effect in the global dimension.

[0086] IP_frequency: the frequency of IP flow control. For example, 1s / 200 / soft means that 200 visits are made within 1 second, which triggers the global soft limit action. During the restricted period, the user's request will be delayed according to a certain algorithm (not directly rejected).

[0087] Read_frequency: read limit frequency (regardless of IP address). For example, 60s / 5000 / 60s / hard means that if the number of reads reaches 5000 times within 60 seconds, the global limit will be triggered and the hard limit will be reached. During the limit period, the request will be rejected.

[0088] Write_frequency: the write throttling frequency (regardless of IP address). For example, 60s / 3000 / 60s / hard means that if the number of writes reaches 3000 times within 60 seconds, the global throttling will be triggered and the hard limit will be reached. During the throttling period, requests will be rejected.

[0089] Throttle: the current limiting triggering algorithm. When the current limiting frequency reaches the configured threshold, there will be a corresponding limiting strategy, which is configured here as: Sliding_window: the sliding window. The single IP 1s / 200 times configured on the user side will be split into 1 window of 100ms in actual internal implementation. For example, the user's access frequency in windows 1-10 is 20, 20, 20, 20, 20, 20, 20, 20, 21, and the cumulative number reaches 201 times (within 1 second). Then, statistics will be taken in window 11. If the number of requests in window 11 does not exceed 19 times, all requests can be released. If it exceeds, the excess part will be delayed or rejected with reference to the configured soft / hard policy.

[0090] User: user dimension. In the figure, two redis users, Peter and Alex, are configured.

[0091] For user Peter, the user dimension is configured with 1,000 accesses within 60 seconds, the restriction policy is hard limit, and the command level is also configured: GET type request 60s / 500 / hard, and SET type request 60s / 500 / hard.

[0092] For user Alex, only 750 accesses are configured within 60 seconds, and the restriction policy is a hard limit.

[0093] Speical: There are two types of IPlists: Blacklist and Whitelist. The corresponding IPlists can be configured. IPs configured in the Whitelist will not be counted by default in the current limiting rules and will be allowed to pass. IPs configured in the Blacklist will not be allowed to access the current limiting rules.

[0094] Through the above configuration, the matching rule engine can be realized, so that the corresponding configuration and code can be separated to a certain extent, and can support certain data operations, logical operations and other functions, and have rule extensibility. For example, when expanding new rules, the program can automatically identify and hot load the rule configuration file.

[0095] Since the original twemproxy is a single-threaded architecture, in order to increase the flow control function, this application sets up multiple threads in the flow statistics aggregation module for adaptation. The specific architecture is as follows Figure 5 As shown, it mainly includes MainThread (MT), Global-Counter-Thread (GCT), User-Counter-Thread (UCT), Delay-Executive-Thread (DET) and Special-List-Thread (SLT).

[0096] Among them, MainThread (MT): the main thread, where the forwarding logic of user requests will be completed, and each user request received will be sent to the Global-Counter-Thread in real time. Global-Counter-Thread (GCT): the current limiting statistics thread of the global dimension. The global configuration items in the current limiting rule configuration will be counted in this thread. User-Counter-Thread (UCT): the current limiting statistics thread of the user dimension. The user configuration items in the current limiting rule configuration will be counted in this thread. Delay-Executive-Thread (DET): the current limiting delayed execution thread. When the request triggers the soft strategy, it will be forwarded to this thread, and the delayed execution will be triggered later. Special-List-Thread (SLT): the special list statistics thread. For the Blacklist and Whitelist configured for statistical current limiting rules.

[0097] Through the multiple threads set up above, the following interactions between threads can be achieved:

[0098] When twemproxy starts loading the corresponding current limiting configuration, SLT loads the Blacklist and Whitelist. If the current limiting configuration changes due to external modifications, the bottom layer will perceive it through the inotify mechanism and update the Blacklist and Whitelist in memory. These two lists will be stored in global variables for GCT and UCT to read, and this global variable will hold a read-write lock to prevent the memory information from being changed or reading the wrong information.

[0099] After receiving the user's request, the MT-main thread will forward the five-tuple of the request, "user name, timestamp, request type, request key, request value", to GCT via UnixSocket.

[0100] After receiving the request from MT, GCT will internally count the number of requests in a window of 100 milliseconds. The requested IP will be maintained in LRUcache mode. IPs that have not been accessed for a long time will be eliminated from the memory, otherwise the IP pool will be too large. After the statistics are completed at the GCT layer, the request will continue to fall to UCT.

[0101] UCT will be split into multiple independent hash tables according to the user dimension. The key name is the user name, and the value will be a structure that includes the access timestamp and specific request type.

[0102] Before GCT and UCT count requests, they will first read the contents of the Blacklist and Whitelist (try to add a read lock). Requests in the Whitelist will be released by default, while requests in the Blacklist will be intercepted and rejected by default.

[0103] DET also maintains a queue. All policies that reach soft will be received by DET, and a random number delay between [0.100, 2.000] seconds will be generated for execution. When the execution time is reached, DET also needs to send the request back to MT, and the final request forwarding logic still needs to be completed in MT.

[0104] After completing the above configuration, this application can determine whether to allow the client to access the Redis cluster based on the passive current limiting method and the access request. The specific implementation process is as follows: first use the current limiting algorithm corresponding to the passive current limiting method to count the client's access traffic, then obtain the traffic IP corresponding to the access traffic, and based on the traffic IP and the first IP list corresponding to the whitelist, determine whether to allow the client to access the Redis cluster.

[0105] Among them, based on the traffic IP and the first IP list corresponding to the whitelist, judging whether to allow the client to access the Redis cluster includes two judgment methods. The first method: if the traffic IP exists in the first IP list, the client is allowed to access the Redis cluster. The second method: if the traffic IP does not exist in the first IP list, judge whether there is a traffic IP in the second IP list corresponding to the blacklist, and if the traffic IP does not exist in the second IP list corresponding to the blacklist, and the access traffic does not reach the global current limiting threshold, judge whether to allow the client to access the Redis cluster based on the traffic type corresponding to the traffic IP.

[0106] There are two ways to determine whether to allow the client to access the Redis cluster based on the traffic type corresponding to the traffic IP. The first way is: if the traffic type corresponding to the traffic IP is write traffic, the traffic IP does not reach the write traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster. The second way is: if the traffic type corresponding to the traffic IP is read traffic, the traffic IP does not reach the read traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster.

[0107] In addition, the above passive current limiting method can be combined with the collaborative current limiting method, which is configured as a switch, and the administrator decides whether to enable it. Among them, collaborative current limiting means that for the same redis cluster, there are multiple twemproxy instances, and twemproxy's current limiting is not divide and conquer, but centralized current limiting.

[0108] When the collaborative current limiting mode is enabled, twemproxy will communicate with each other through the rpc protocol about its own IP request frequency within the unit period, the request frequency in the user dimension, etc., to ensure that the current limiting of the same IP or user will take effect globally.

[0109] For example, if the IP address 10.10.9.1 is accessed more than 1,000 times within 30 seconds and current limiting is triggered, it will take effect in all twemproxy instances. That is, the current limiting counter is not limited to the memory of a single twemproxy instance, but will be "broadcasted" to all twemproxy proxy levels, thereby improving the accuracy of current limiting.

[0110] Further, combined with Figure 6 The specific implementation process of determining whether to allow the client to access the Redis cluster when the traffic limiting mode is the collaborative limiting mode is described. Specifically, it is necessary to first determine whether to enable the collaborative limiting mode, obtain the determination result, and then determine whether to allow the client to access the Redis cluster based on the determination result.

[0111] Among them, based on the judgment result, to determine whether the client is allowed to access the Redis cluster, the following judgment needs to be made, that is, if the judgment result is no, the client is allowed to access the Redis cluster, if the judgment result is yes, the current limiting algorithm corresponding to the collaborative current limiting method is used to count the frequency corresponding to the access request, and then based on the frequency and the global current limiting threshold, determine whether the client is allowed to access the Redis cluster. Among them, based on the frequency and the global current limiting threshold, to determine whether the client is allowed to access the Redis cluster, it is mainly to determine whether the frequency reaches the threshold, that is, if the frequency does not reach the global current limiting threshold, the client is allowed to access the Redis cluster. On the contrary, if the frequency reaches the global current limiting threshold, the client is not allowed to access the Redis cluster.

[0112] The active current limiting method is different from the passive current limiting method. The core factor considered in the active current limiting method is no longer the refined user request statistics. The main factors considered are: twemproxy process CPU utilization, user QPS, the number of slow queries per unit time, the number of batch query requests per unit time, whether the cluster is performing master-slave switching, etc.

[0113] The above factors are combined to determine whether the active current limiting method needs to be triggered. Once the active current limiting method is triggered, the passive current limiting method will be temporarily closed during its validity period (except for the Whitelist and Blacklist).

[0114] Combine the following Figure 7The implementation process of judging whether to allow the client to access the Redis cluster based on the active flow limiting mode and the access request when the flow limiting mode is the active flow limiting mode is described. Specifically, the flow limiting algorithm corresponding to the active flow limiting mode is first used to process the information in the access request, and the conditional data of the flow IP corresponding to the access request is screened, wherein the conditional data at least includes CPU utilization, client QPS, number of slow queries, number of batch requests, and data used to characterize whether to perform master-slave switching. Then, based on the conditional data of the flow IP and the preset judgment rules, it is judged whether to allow the client to access the Redis cluster.

[0115] Among them, there will be a separate thread in Twemproxy, which will perform various data statistics: it will obtain the CPU utilization of the corresponding twemproxy process in the system / proc / pid / stat every second, aggregate the number of user QPS once a second, obtain the current number of slow queries of twemproxy once every 5 seconds, and obtain the current number of batch requests once a second. This type is mainly for redis commands such as hgetall, mget, hget, and determines whether the corresponding redis cluster is executing master-slave switching.

[0116] The CPU and QPS of the process are used as the core judgment conditions. When the growth of CPU is linearly related to the growth of QPS, and the CPU of the process is greater than 97% in 10 cycles out of 15 consecutive cycles (1 cycle per second), the token bucket algorithm will be enabled internally for current limiting: a portion of "tokens" will be placed in the bucket by default, and there will be a total of 4 interactions between the user, twemproxy and redis. When twemproxy receives a user request, twemproxy forwards the request to redis, redis returns the result to twemproxy, and twemproxy returns the result to the user. When twemproxy receives a user request, it is considered to have taken away the "token". When twemproxy replies the request result to the user, it is considered to have put the "token" back. If there is always a token, the user request will not be interrupted, but once the token is taken out, subsequent requests will be directly rejected.

[0117] The number of slow queries per unit time and the number of batch queries per unit time are used as secondary judgment conditions. When the core judgment conditions are not met, the secondary conditions are judged again: for example, if the process CPU utilization is between 60% and 90%, the number of slow queries per second and the number of batch queries will be judged. If the number of slow queries per second is greater than 30 or the number of batch queries is greater than 100, and 10 / 15 is met (each 1 second is a cycle, and 10 cycles can be met in 15 consecutive cycles), the time consumption of the currently executing statement will be analyzed. For those with too long execution time (such as the request has been sent to redis, but the result has not been returned), twemproxy will analyze the source IP of these requests, and then enable the token bucket algorithm for current limiting.

[0118] There is also a special scenario, that is, the redis cluster switches from master to slave normally. At this time, the CPU pressure of the new redis master instance will increase. While taking on user traffic, it also needs to synchronize data with other slaves. If a large amount of user traffic is updated and written during the period when the master-slave synchronization is not completed, it may trigger the failure of the master-slave synchronization, thus causing a vicious cycle. Therefore, during the master-slave switching, twemproxy will also determine the number of writes and the size of the write traffic in this cycle. When it exceeds 40 per second, the token bucket algorithm will be used to limit the flow of slow-check or batch-query user IPs.

[0119] Therefore, based on the above reasons, it is determined whether the client is allowed to access the Redis cluster based on the conditional data of the traffic IP and the preset judgment rules, wherein the determination of allowing the client to access the Redis cluster is made in the following manner: when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization rate meets the preset utilization rate, and the traffic IP does not reach the batch request threshold, the client is allowed to access the Redis cluster; and when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, and the CPU utilization rate does not meet the preset utilization rate, the client is allowed to access the Redis cluster;

[0120] The following method is used to judge whether the client is not allowed to access the Redis cluster: when the data used to characterize whether the master-slave switch is executed is that the master-slave switch is executed, the client is not allowed to access the Redis cluster; when the data used to characterize whether the master-slave switch is executed is that the master-slave switch is not executed and the traffic IP does not reach the write traffic threshold, the client is not allowed to access the Redis cluster; when the data used to characterize whether the master-slave switch is executed is that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, and the traffic IP does not reach the slow query threshold, the client is not allowed to access the Redis cluster; when the data used to characterize whether the master-slave switch is executed is that the master-slave switch is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP reaches the batch request threshold, the client is not allowed to access the Redis cluster.

[0121] The embodiment of the present invention provides an access control method, including: receiving an access request from a client, then determining a flow restriction method corresponding to the client based on the access request, and then judging whether to allow the client to access a Redis cluster based on the flow restriction method and the access request. The present invention sets an access control policy at the twemproxy proxy middleware layer, that is, identifies the received access request, and selectively intercepts it using a corresponding flow restriction method, thereby avoiding overload of the entire Redis server cluster and ensuring the stability of the Redis server cluster.

[0122] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0123] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0124] Figure 8 The schematic diagram of the structure of an access control device provided by an embodiment of the present invention is shown. For the convenience of explanation, only the part related to the embodiment of the present invention is shown. An access control device includes a receiving module 81, a current limiting mode determination module 82 and an access control module 83, which are as follows:

[0125] A receiving module 81, used for receiving an access request from a client;

[0126] A flow limiting mode determination module 82 is used to determine a flow limiting mode corresponding to the client based on the access request;

[0127] The access control module 83 is used to determine whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request.

[0128] In a possible implementation, the flow limiting method is a passive flow limiting method;

[0129] The access control module 83 is also used to use the current limiting algorithm corresponding to the passive current limiting method to count the client's access traffic; obtain the traffic IP corresponding to the access traffic; and determine whether to allow the client to access the Redis cluster based on the traffic IP and the first IP list corresponding to the whitelist.

[0130] In a possible implementation, the access control module 83 is further configured to allow the client to access the Redis cluster if there is a traffic IP in the first IP list.

[0131] In one possible implementation, the access control module 83 is also used to determine whether there is a traffic IP in the second IP list corresponding to the blacklist if there is no traffic IP in the first IP list; if there is no traffic IP in the second IP list corresponding to the blacklist, and the access traffic does not reach the global flow limiting threshold, based on the traffic type corresponding to the traffic IP, determine whether to allow the client to access the Redis cluster.

[0132] In a possible implementation, the access control module 83 is also used to allow the client to access the Redis cluster if the traffic type corresponding to the traffic IP is write traffic, the traffic IP does not reach the write traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client.

[0133] In a possible implementation, the access control module 83 is also used to allow the client to access the Redis cluster if the traffic type corresponding to the traffic IP is read traffic, the traffic IP does not reach the read traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client.

[0134] In a possible implementation, the device also includes a collaborative current limiting module, which is used to determine whether to enable the collaborative current limiting mode and obtain a determination result; based on the determination result, determine whether to allow the client to access the Redis cluster.

[0135] In one possible implementation, if the judgment result of the collaborative current limiting module is no, the client is allowed to access the Redis cluster; if the judgment result is yes, the current limiting algorithm corresponding to the collaborative current limiting method is used to count the frequency corresponding to the access request; based on the frequency and the global current limiting threshold, it is determined whether the client is allowed to access the Redis cluster.

[0136] In a possible implementation, if the frequency does not reach the global current limiting threshold, the collaborative current limiting module allows the client to access the Redis cluster.

[0137] In a possible implementation, the flow limiting mode is an active flow limiting mode;

[0138] The access control module 83 is also used to process the information in the access request using the current limiting algorithm corresponding to the active current limiting method, and to screen the conditional data of the traffic IP corresponding to the access request, wherein the conditional data includes at least CPU utilization, client QPS, number of slow queries, number of batch requests, and data used to characterize whether to perform master-slave switching; based on the conditional data of the traffic IP and preset judgment rules, determine whether to allow the client to access the Redis cluster.

[0139] In a possible implementation, the access control module 83 is also used to allow the client to access the Redis cluster when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization and the traffic IP does not reach the batch request threshold; the client is allowed to access the Redis cluster when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, and the CPU utilization does not meet the preset utilization.

[0140] In a possible implementation, the access control module 83 is also used to not allow the client to access the Redis cluster when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is executed; when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed and the traffic IP does not reach the write traffic threshold, the client is not allowed to access the Redis cluster; when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, and the traffic IP does not reach the slow query threshold, the client is not allowed to access the Redis cluster; when the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP reaches the batch request threshold, the client is not allowed to access the Redis cluster.

[0141] Fig. 9 is a schematic diagram of a terminal provided by an embodiment of the present invention. Fig. 9 As shown, the terminal 9 of this embodiment includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91. When the processor 91 executes the computer program 93, the steps in the above-mentioned various access control method embodiments are implemented, for example Figure 1 Alternatively, when the processor 91 executes the computer program 93, the functions of each module / unit in the above-mentioned access control device embodiments are realized, for example Figure 8 Functions of modules / units 81 to 83 are shown.

[0142] The present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the access control methods provided by the various implementation modes described above.

[0143] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0144] The present invention also provides a program product, which includes an execution instruction stored in a readable storage medium. At least one processor of a device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the access control method provided by the various embodiments described above.

[0145] In the above-mentioned device embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly implemented as being executed by a hardware processor, or may be implemented as being executed by a combination of hardware and software modules in the processor.

[0146] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An access control method, characterized in that: include: Receive access requests from clients; Based on the access request, determine a flow restriction method corresponding to the client; Based on the traffic restriction method and the access request, determining whether to allow the client to access the Redis cluster; The flow limiting method is a passive flow limiting method; The determining whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request includes: Using the current limiting algorithm corresponding to the passive current limiting method to count the access traffic of the client; Obtain the traffic IP corresponding to the access traffic; Based on the traffic IP and the first IP list corresponding to the whitelist, determine whether to allow the client to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the traffic IP and the first IP list corresponding to the whitelist includes: If the traffic IP exists in the first IP list, the client is allowed to access the Redis cluster; If the traffic IP does not exist in the first IP list, determine whether the traffic IP exists in the second IP list corresponding to the blacklist; If the traffic IP does not exist in the second IP list corresponding to the blacklist, and the access traffic does not reach the global current limiting threshold, based on the traffic type corresponding to the traffic IP, determine whether to allow the client to access the Redis cluster; The flow limiting mode is an active flow limiting mode; The determining whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request includes: The information in the access request is processed using a current limiting algorithm corresponding to the active current limiting method, and conditional data of the traffic IP corresponding to the access request is screened, wherein the conditional data at least includes CPU utilization, client QPS, number of slow queries, number of batch requests, and data used to characterize whether to perform master-slave switching; Based on the conditional data of the traffic IP and the preset determination rules, determine whether to allow the client to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the conditional data of the traffic IP and a preset determination rule includes: When the data used to characterize whether to perform master-slave switching is that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP does not reach the batch request threshold, the client is allowed to access the Redis cluster; When the data used to characterize whether to perform master-slave switching is that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, and the CPU utilization does not meet the preset utilization, the client is allowed to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the conditional data of the traffic IP and a preset determination rule includes: In a case where the data used to indicate whether to perform master-slave switching indicates that master-slave switching is to be performed, the client is not allowed to access the Redis cluster; If the data used to characterize whether the master-slave switching is performed indicates that the master-slave switching is not performed and the traffic IP does not reach the write traffic threshold, the client is not allowed to access the Redis cluster; If the data used to characterize whether to perform master-slave switching indicates that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, and the traffic IP does not reach the slow query threshold, the client is not allowed to access the Redis cluster; When the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization and the traffic IP reaches the batch request threshold, the client is not allowed to access the Redis cluster.

2. The access control method according to claim 1, characterized in that: The determining whether to allow the client to access the Redis cluster based on the traffic type corresponding to the traffic IP includes: If the traffic type corresponding to the traffic IP is write traffic, the traffic IP does not reach the write traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster.

3. The access control method according to claim 1, characterized in that: The determining whether to allow the client to access the Redis cluster based on the traffic type corresponding to the traffic IP includes: If the traffic type corresponding to the traffic IP is read traffic, the traffic IP does not reach the read traffic threshold, and the traffic IP reaches the access traffic threshold corresponding to the client, the client is allowed to access the Redis cluster.

4. The access control method according to claim 1, characterized in that: The method further comprises: Determine whether to enable the collaborative current limiting mode and obtain the determination result; Based on the determination result, determine whether to allow the client to access the Redis cluster.

5. The access control method according to claim 4, characterized in that: The determining, based on the determination result, whether to allow the client to access the Redis cluster includes: If the judgment result is no, allowing the client to access the Redis cluster; If the judgment result is yes, the frequency corresponding to the access request is counted using a current limiting algorithm corresponding to the collaborative current limiting method; Based on the frequency and the global current limiting threshold, determine whether to allow the client to access the Redis cluster.

6. The access control method according to claim 5, characterized in that: The determining whether to allow the client to access the Redis cluster based on the frequency and the global current limiting threshold includes: If the frequency does not reach the global current limiting threshold, the client is allowed to access the Redis cluster.

7. An access control device, characterized in that: include: A receiving module, used for receiving an access request from a client; A flow limiting mode determination module, used to determine a flow limiting mode corresponding to the client based on the access request; An access control module, used to determine whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request; The flow limiting method is a passive flow limiting method; The determining whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request includes: Using the current limiting algorithm corresponding to the passive current limiting method to count the access traffic of the client; Obtain the traffic IP corresponding to the access traffic; Based on the traffic IP and the first IP list corresponding to the whitelist, determine whether to allow the client to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the traffic IP and the first IP list corresponding to the whitelist includes: If the traffic IP exists in the first IP list, the client is allowed to access the Redis cluster; If the traffic IP does not exist in the first IP list, determine whether the traffic IP exists in the second IP list corresponding to the blacklist; If the traffic IP does not exist in the second IP list corresponding to the blacklist, and the access traffic does not reach the global current limiting threshold, based on the traffic type corresponding to the traffic IP, determine whether to allow the client to access the Redis cluster; The flow limiting mode is an active flow limiting mode; The determining whether to allow the client to access the Redis cluster based on the traffic restriction method and the access request includes: The information in the access request is processed using a current limiting algorithm corresponding to the active current limiting method, and conditional data of the traffic IP corresponding to the access request is screened, wherein the conditional data at least includes CPU utilization, client QPS, number of slow queries, number of batch requests, and data used to characterize whether to perform master-slave switching; Based on the conditional data of the traffic IP and the preset determination rules, determine whether to allow the client to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the conditional data of the traffic IP and a preset determination rule includes: When the data used to characterize whether to perform master-slave switching is that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization, and the traffic IP does not reach the batch request threshold, the client is allowed to access the Redis cluster; When the data used to characterize whether to perform master-slave switching is that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, and the CPU utilization does not meet the preset utilization, the client is allowed to access the Redis cluster; The determining whether to allow the client to access the Redis cluster based on the conditional data of the traffic IP and a preset determination rule includes: In a case where the data used to indicate whether to perform master-slave switching indicates that master-slave switching is to be performed, the client is not allowed to access the Redis cluster; If the data used to characterize whether the master-slave switching is performed indicates that the master-slave switching is not performed and the traffic IP does not reach the write traffic threshold, the client is not allowed to access the Redis cluster; If the data used to characterize whether to perform master-slave switching indicates that master-slave switching is not performed, the traffic IP does not reach the write traffic threshold, and the traffic IP does not reach the slow query threshold, the client is not allowed to access the Redis cluster; When the data used to characterize whether the master-slave switching is executed is that the master-slave switching is not executed, the traffic IP does not reach the write traffic threshold, the traffic IP does not reach the slow query threshold, the CPU utilization meets the preset utilization and the traffic IP reaches the batch request threshold, the client is not allowed to access the Redis cluster.

8. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the access control method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the access control method according to any one of claims 1 to 6 are implemented.

Images