Distributed database flow limiting method and related equipment
By determining whether an SQL statement meets the rate limiting conditions based on rate limiting rules in a distributed database and executing the operation in the monitoring record, the problem of poor stability and adaptability of existing rate limiting methods is solved, achieving higher system scalability and stability.
Patent Information
- Application Number
- CN202511020941.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-23
- Publication Date
- 2025-11-11
AI Technical Summary
Existing distributed database rate limiting methods suffer from poor stability and adaptability. In particular, rate limiting schemes at the computing and sharded storage layers fail to meet users' actual needs, and their reliance on centralized components affects system scalability and stability.
By obtaining the target user's SQL statement, determining the second rate limiting rule based on the first rate limiting rule, judging whether the SQL statement meets the rate limiting rule, and executing the rate limiting operation in the monitoring record, fine-grained rate limiting control is achieved, avoiding reliance on centralized components and improving system scalability and stability.
It enables rate limiting operations in distributed databases without relying on centralized components, improving the system's scalability and stability, making it more adaptable, and able to accurately filter SQL statements that do not require rate limiting, thus providing a better user experience.
Smart Images
Figure CN120929477A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer technology, and in particular to a distributed database rate limiting method, a distributed database rate limiting device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology
[0002] For distributed databases with separate storage and computation, a distributed Structured Query Language (SQL) agent is often used as the computation service layer to provide user access and database services.
[0003] In related technologies, rate limiting for user queries can be achieved through SQL matching and SQL fingerprinting, which can be implemented at the computation layer or the sharded storage layer. However, these rate limiting methods suffer from poor stability and adaptability.
[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] This disclosure provides a distributed database rate limiting method and related equipment, which at least to some extent overcomes the problems of poor stability and poor versatility of distributed database rate limiting methods in related technologies.
[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.
[0007] According to one aspect of this disclosure, a distributed database rate limiting method is provided, comprising: obtaining a structured query language (SQL) statement of a target user; if the target user satisfies a first rate limiting rule, determining a second rate limiting rule for the SQL statement based on the first rate limiting rule; if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in a monitoring record, then performing a corresponding rate limiting operation.
[0008] In one embodiment of this disclosure, the method further includes: if the SQL statement satisfies the second rate limiting rule and the SQL statement is not in the monitoring record, then performing SQL processing with rate limiting detection.
[0009] In one embodiment of this disclosure, the method further includes: if the execution time of the SQL statement meets the preset SQL rate limiting condition, then generating a monitoring record of the SQL statement according to the rate limiting operation configured in the second rate limiting rule; storing the monitoring record of the SQL statement in the target shard of the current computing node, so that other computing nodes in the distributed database can pull the monitoring record of the SQL statement.
[0010] In one embodiment of this disclosure, the preset SQL rate limiting condition is configured in the second rate limiting rule, and the preset SQL rate limiting condition includes a threshold for the execution time of the SQL statement that triggers rate limiting; wherein, the method further includes: if the execution time of the SQL statement reaches the threshold for the execution time of the SQL statement that triggers rate limiting, then it is determined that the execution time of the SQL statement satisfies the preset SQL rate limiting condition.
[0011] In one embodiment of this disclosure, the second rate limiting rule includes at least one of the following: rate limiting rule name, rate limiting operation after triggering rate limiting, continuous effective duration threshold after triggering rate limiting, priority of rate limiting rule, threshold of SQL statement execution time that triggers rate limiting, and SQL matching pattern.
[0012] In one embodiment of this disclosure, the first rate limiting rule includes at least one of the following: the keyword of the SQL statement associated with the second rate limiting rule, the user to be matched for SQL rate limiting, and the address of the user to be matched for SQL rate limiting.
[0013] In one embodiment of this disclosure, after obtaining the SQL statement of the target user, the method further includes: if the target user exists in a preset whitelist, or if the target user is not in the preset whitelist and there is no first rate limiting rule and the second rate limiting rule, then the SQL statement is executed to obtain a query result, and the query result is sent to the target user.
[0014] In one embodiment of this disclosure, after performing the corresponding rate limiting operation if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record, the method further includes: if the buffer is not full, then writing the SQL statements that satisfy the second rate limiting rule into the buffer in batches; if the buffer is full or a timed flush task is performed, then writing the SQL statements that satisfy the second rate limiting rule from the buffer into the database.
[0015] According to another aspect of this disclosure, a distributed database rate limiting device is also provided, comprising: a statement acquisition module for acquiring SQL statements of a target user; a rule determination module for determining a second rate limiting rule for the SQL statement based on the first rate limiting rule if the target user satisfies a first rate limiting rule; and a rate limiting execution module for executing a corresponding rate limiting operation if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in a monitoring record.
[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described distributed database rate limiting method by executing the executable instructions.
[0017] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described distributed database rate limiting method.
[0018] According to another aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program or computer instructions, the computer program or computer instructions being loaded and executed by a processor to enable a computer to implement the above-described distributed database rate limiting method.
[0019] In this embodiment, the SQL statement of the target user is obtained; if the target user meets the first rate limiting rule, the second rate limiting rule of the SQL statement is determined according to the first rate limiting rule; if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, the corresponding rate limiting operation is executed. This disclosure determines whether to execute the rate limiting operation based on the judgment result by judging whether the SQL statement meets the rate limiting rule and whether the SQL statement is already in the monitoring record, thereby improving the scalability and stability of the system without relying on centralized components.
[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0022] Figure 1The flowchart illustrates a distributed database rate limiting method provided in an embodiment of this disclosure.
[0023] Figure 2 A flowchart of another distributed database rate limiting method provided in an embodiment of this disclosure is shown.
[0024] Figure 3 A flowchart of a monitoring record generation method provided in an embodiment of this disclosure is shown.
[0025] Figure 4 This diagram illustrates a flowchart of another distributed database rate limiting method provided in an embodiment of this disclosure.
[0026] Figure 5 The flowchart illustrates another distributed database rate limiting method provided in an embodiment of this disclosure.
[0027] Figure 6 The following is a flowchart illustrating an example of a distributed database rate limiting method provided in this disclosure.
[0028] Figure 7 The flowchart illustrates a second example of a distributed database rate limiting method provided in this disclosure.
[0029] Figure 8 The following is a flowchart of Example 3 of a distributed database rate limiting method provided in this disclosure.
[0030] Figure 9 This diagram illustrates the structure of a distributed database rate limiting device provided in an embodiment of this disclosure.
[0031] Figure 10 A structural block diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation
[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0033] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the following detailed description is provided in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.
[0035] For distributed databases with separate storage and compute layers, a distributed Structured Query Language (SQL) agent is typically used as the compute service layer to provide user access and database services. The storage layer is generally divided into several shards, with a single-machine database instance deployed on each shard. Data is stored on different shards according to certain distribution rules. The number of shards and the distribution rules in the storage layer can be determined based on actual needs.
[0036] When a user issues an SQL query command, the computation layer parses and optimizes the SQL query command according to the connection protocol, generates a corresponding execution plan, executes the execution plan through the SQL execution engine, summarizes the query results returned from each shard, and performs subsequent calculations, such as aggregation, sorting, and filtering. Finally, the final calculation result is returned to the user.
[0037] In related technologies, rate limiting schemes for distributed databases typically use SQL matching or SQL fingerprinting to limit user queries. There are generally two solutions: one implemented at the computation layer, and the other implemented at the sharded storage layer.
[0038] The architecture implemented in the computing layer includes: the front-end application sends the user's query request, the computing node receives the query request, and performs rate limiting rule matching. If the match is successful, the rate limiting is returned; if the match fails, the SQL is executed.
[0039] Rate limiting solutions for sharded storage layers are typically implemented using the following methods: rate limiting implemented by the database itself, rate limiting implemented through proxy tools, and rate limiting on the business side.
[0040] The rate limiting scheme implemented by the database itself is similar to the architecture of the computing layer. However, the rate limiting function provided by the database itself is limited and cannot perform fuzzy matching based on the parameterized SQL statements, making it difficult to meet the actual needs of users.
[0041] Rate limiting solutions implemented through proxy tools are usually used when the rate limiting provided by the database itself cannot meet business needs. A layer of SQL proxy is added to the front end of the sharding. The SQL proxy implements the rate limiting function for SQL, such as ProxySQL.
[0042] The business-side rate limiting solution arises because neither of the above two methods meets the business requirements. For example, in MySQL, the corresponding process includes: connecting to the MySQL database, creating a counter table, setting the initial value of the counter table, checking the counter before performing database operations, updating the counter, performing database operations, updating the counter again, and disconnecting the MySQL database connection.
[0043] Rate limiting solutions implemented by the database itself have limited effectiveness and cannot meet the actual needs of users. Rate limiting solutions implemented by proxy tools require additional network communication, which reduces service stability, while business-side rate limiting solutions require additional business configuration and lack universality.
[0044] The rate limiting scheme of the sharded storage layer only limits the rate of a single operator in the confidence plan, and cannot limit the rate of SQL operations of the entire distributed system.
[0045] Distributed rate limiting schemes typically only limit the queries per second (QPS), while also needing to handle coordination and consistency among multiple computing nodes, which increases the complexity of the system. Distributed rate limiting strategies usually rely on centralized components, which may become a single point of failure in the system, affecting the scalability and stability of the system.
[0046] To at least partially solve the aforementioned technical problems, this disclosure obtains the target user's SQL statement; if the target user meets the first rate limiting rule, then a second rate limiting rule for the SQL statement is determined based on the first rate limiting rule; if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, the corresponding rate limiting operation is executed. This disclosure determines whether to execute a rate limiting operation based on the judgment result of whether the SQL statement meets the rate limiting rule and whether the SQL statement is already in the monitoring record, thereby improving the scalability and stability of the system without relying on centralized components.
[0047] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.
[0048] The following detailed description of this exemplary implementation method is provided in conjunction with the accompanying drawings and embodiments.
[0049] First, this disclosure provides a distributed database rate limiting method, which can be executed by any electronic device with computing capabilities. The distributed database rate limiting method provided in this disclosure can be executed by the current computing node, for example, by a distributed database rate limiting device configured on the current computing node. This device can be a Programmable Logic Controller (PLC) or a microcontroller, as long as it can perform the corresponding functions.
[0050] Figure 1 This diagram illustrates a flowchart of a distributed database rate limiting method provided in an embodiment of this disclosure, such as... Figure 1 As shown in the embodiments of this disclosure, the distributed database rate limiting method is applied to the current computing node, and the method mainly includes the following steps:
[0051] S102. Obtain the SQL statement of the target user.
[0052] In one embodiment, the target user is the user who initiates the SQL statement. The target user can initiate the SQL statement using a corresponding terminal device, which has an application client installed. The terminal device can be various electronic devices, including but not limited to smartphones, tablets, smartwatches, wearable devices, etc.
[0053] SQL statements can include SQL statements corresponding to data query requests, data insertion requests, data update requests, or data deletion requests. SQL statements can include at least one of the following: table name, column names retrieved from the table, query conditions, sorting conditions, and grouping conditions.
[0054] For example, the SQL statement "SELECT id, name FROM users WHERE age>25 LIMIT 10" can filter 10 entries from the users table whose id and name are greater than 25.
[0055] After the current computing node obtains the target user's SQL statement, it can perform query parsing on the SQL statement. The SQL parser checks whether the SQL statement's syntax is correct. If the syntax is correct, the SQL statement is converted into an internal data structure.
[0056] S104. If the target user meets the first rate limiting rule, then determine the second rate limiting rule for the SQL statement based on the first rate limiting rule.
[0057] In one embodiment, the first rate limiting rule is used to limit the number of users who need SQL rate limiting; the second rate limiting rule is used to limit the parameters of SQL statements.
[0058] The first rate limiting rule may include at least one of the following: the keyword of the SQL statement that associates with the second rate limiting rule, the user to be matched for SQL rate limiting, and the address of the user to be matched for SQL rate limiting.
[0059] The keywords in the SQL statement that associate the second rate limiting rule are used to determine the correspondence between the first and second rate limiting rules. The keywords in the SQL statement that associate the second rate limiting rule can include the name of the rate limiting rule in the second rate limiting rule.
[0060] The target user satisfies the first rate limiting rule if the target user's username is the same as the value of the user field that needs to be matched for SQL rate limiting in the first rate limiting rule, and / or the target user's address is the same as the value of the user address field that needs to be matched for SQL rate limiting in the first rate limiting rule.
[0061] In one embodiment, the second rate limiting rule may include at least one of the following: rate limiting rule name, rate limiting operation after triggering rate limiting, threshold for the duration of continuous effect after triggering rate limiting, priority of rate limiting rule, threshold for the execution time of the SQL statement that triggers rate limiting, and SQL matching pattern.
[0062] The name of the rate limiting rule is a unique identifier for the rate limiting rule, such as rulename1.
[0063] The rate limiting action after triggering rate limiting can support Kill mode, Low Priority mode, Dry Run mode, etc. The rate limiting action corresponding to Kill mode is to close the current query connection, the rate limiting action corresponding to Low Priority mode is to reduce the execution priority of the SQL statement, and the rate limiting action corresponding to Dry Run mode is to do nothing.
[0064] The duration threshold for the continued effectiveness of rate limiting after it is triggered refers to the duration of the rate limiting operation after it is triggered. For example, duration = 1000.
[0065] The priority of a rate limiting rule can be set so that when a second rate limiting rule is matched, the matching will be performed according to the priority configured in the second rate limiting rule. For example, Priority=100.
[0066] The threshold for the execution time of the SQL statement that triggers rate limiting, exec_elapsed, refers to the time interval at which the rate limiting operation is triggered when the second rate limiting rule is met again after the execution time threshold has elapsed since the first time the second rate limiting rule was met. For example, exec_elapsed = 1000.
[0067] SQL matching modes support Exact, Similar, ExecPlan, etc. Exact is for exact SQL matching, Similar is for parameterized SQL matching, and ExecPlan is for execution plan matching.
[0068] In one embodiment, if the duration of the SQL statement that triggers rate limiting reaches the threshold for the duration of continuous effectiveness after the rate limiting is triggered, the rate limiting operation on the SQL statement is lifted.
[0069] In one embodiment, for a user who needs rate limiting, a first rate limiting rule and multiple second rate limiting rules can be configured for the target user, with the multiple second rate limiting rules configured with different priorities.
[0070] When the target user meets the first rate limiting rule, multiple second rate limiting rules corresponding to the SQL statement can be determined based on the keywords in the first rate limiting rule. The second rate limiting rule with the highest priority among the multiple second rate limiting rules is determined as the second rate limiting rule for the SQL statement.
[0071] This disclosure enables fine-grained rate limiting control by setting a first rate limiting rule and a second rate limiting rule. Different second rate limiting rules can be configured for SQL statements, thereby improving the adaptability of the rate limiting method.
[0072] S106. If the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, then the corresponding rate limiting operation will be executed.
[0073] The SQL statement satisfying the second rate limiting rule means that the execution time of the SQL statement meets the duration of the rate limiting effect after it is triggered, as configured in the second rate limiting rule.
[0074] The monitoring log is used to monitor SQL statements that remain in effect for the duration after the rate limiting is triggered as configured in the second rate limiting rule.
[0075] Monitoring logs can be stored in the metadata table of the first shard of the current compute node.
[0076] In this embodiment, the current computing node obtains the SQL statement of the target user; if the target user meets the first rate limiting rule, the second rate limiting rule of the SQL statement is determined according to the first rate limiting rule; if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, the corresponding rate limiting operation is executed. This disclosure determines whether to execute the rate limiting operation based on the judgment result of whether the SQL statement meets the rate limiting rule and whether the SQL statement is already in the monitoring record, thereby improving the scalability and stability of the system without relying on centralized components.
[0077] Figure 2 This diagram illustrates a flowchart of another distributed database rate limiting method provided in an embodiment of this disclosure. Figure 1 Based on the previous example, S108 is added after S104 to limit the scenarios where the SQL statement is not in the monitoring log. For example... Figure 2 As shown, in one embodiment, the distributed database rate limiting method of this disclosure includes S102 to S108, wherein:
[0078] S108. If the SQL statement satisfies the second rate limiting rule and the SQL statement is not in the monitoring record, then execute the SQL processing with rate limiting detection.
[0079] If an SQL statement satisfies the second rate limiting rule and is not in the monitoring log, it indicates that when the SQL statement satisfies the second rate limiting rule, the execution time of the SQL statement does not reach the threshold of the SQL statement execution time that triggers rate limiting configured in the second rate limiting rule.
[0080] When an SQL statement meets the second rate-limiting rule, the current compute node can create a rate-limiting detection module. If the SQL statement is not in the monitoring log, then SQL processing with rate-limiting detection is executed. SQL processing with rate-limiting detection includes: adding a rate-limiting detection module to detect the execution time of the SQL statement, executing the SQL statement, obtaining the query results, and sending the query results to the target user.
[0081] In this embodiment of the disclosure, when an SQL statement meets the second rate limiting rule and the SQL statement is not in the monitoring record, by executing SQL processing with rate limiting detection, the rate limiting firewall capability based on the execution time of the SQL statement can be realized without introducing a centralized component.
[0082] Figure 3 A flowchart illustrating a monitoring record generation method provided in an embodiment of this disclosure is shown. Figure 3 As shown, in one embodiment, the distributed database rate limiting method provided in this disclosure further includes:
[0083] S302. If the execution time of the SQL statement meets the preset SQL rate limiting conditions, then a monitoring record of the SQL statement is generated according to the rate limiting operation configured in the second rate limiting rule.
[0084] S304. Store the monitoring records of the SQL statements in the target shard of the current computing node so that other computing nodes in the distributed database can pull the monitoring records of the SQL statements.
[0085] In S302, the preset SQL rate limiting conditions are configured in the second rate limiting rule. The preset SQL rate limiting conditions include a threshold for the execution time of the SQL statement that triggers rate limiting. The value of this threshold can be determined according to actual needs, and this disclosure does not make specific limitations on it.
[0086] In one embodiment, if the execution time of an SQL statement reaches the threshold of the SQL statement execution time that triggers rate limiting, then the execution time of the SQL statement is determined to meet the preset SQL rate limiting conditions.
[0087] The monitoring log for the SQL statement can include the SQL statement, the rate limiting operation in the second rate limiting rule, the name of the rate limiting rule, etc.
[0088] In S304, the monitoring log primarily enables the synchronization of monitoring information between different computing nodes. Other computing nodes in the distributed database refer to computing nodes in the distributed database other than the current computing node.
[0089] After the current compute node generates a monitoring record of the SQL statement, it saves the monitoring record to the metadata table of the target shard. Other compute nodes in the distributed database can periodically pull the metadata table and synchronize the monitoring record of the SQL statement pulled to the memory of their own compute nodes.
[0090] In one embodiment, other computing nodes in the distributed database can also subscribe to the monitoring record update messages of the target shard of the current computing node. When the metadata table of the target shard updates the monitoring record, the current computing node will synchronize the updated monitoring record to other computing nodes.
[0091] The target shard mentioned above can be the first shard in the current computing node, the last shard in the current computing node, or any shard in the current computing node. This disclosure does not make any specific limitation in this regard.
[0092] In the embodiments of this disclosure, by using rate limiting monitoring records, the synchronization of monitoring records between computing nodes can be achieved without introducing external centralized components, thus ensuring the consistency of information between different computing nodes.
[0093] Figure 4This diagram illustrates a flowchart of yet another distributed database rate limiting method provided in an embodiment of this disclosure. Figure 1 Based on the previous embodiment, S103 is added after S102 to limit the scenarios of passing through the whitelist and the existence of rate-limiting rules. For example... Figure 4 As shown, in one embodiment, the distributed database rate limiting method provided by this disclosure includes S102, S103 to S106, wherein:
[0094] S103. If the target user exists in the preset whitelist information, or if the target user does not exist in the preset whitelist information and there is no first rate limiting rule and no second rate limiting rule, then execute the SQL statement, obtain the query result, and send the query result to the target user.
[0095] Pre-configured whitelist information can be configured in the current compute node. This whitelist information can be described using whitelist SQL syntax and may include at least one of the following: accounts in the rate-limiting whitelist and source addresses in the rate-limiting whitelist. The accounts in the rate-limiting whitelist can also be the names of the rate-limiting whitelists. If the account field is configured, only users listed in this field can ignore the first and second rate-limiting rules. Similarly, if the source address field is configured, only client addresses listed in this field can ignore the first and second rate-limiting rules.
[0096] In one embodiment, if the name of the target user does not match the keyword of the second rate limiting rule associated with the SQL statement of the first rate limiting rule stored in the current compute node, it is determined that no first rate limiting rule exists; if the name of the target user matches the keyword of the second rate limiting rule associated with the SQL statement of the first rate limiting rule stored in the current compute node, it is determined that a first rate limiting rule exists for the target user.
[0097] If a first rate-limiting rule exists for the target user, the existence of a second rate-limiting rule can be confirmed by associating the keyword of the first rate-limiting rule with the name of the second rate-limiting rule's rate-limiting rule. If the keyword in the first rate-limiting rule's SQL statement associated with the second rate-limiting rule contains the name of the second rate-limiting rule's rate-limiting rule, then a second rate-limiting rule exists. If the keyword in the first rate-limiting rule's SQL statement associated with the second rate-limiting rule does not contain the name of the second rate-limiting rule's rate-limiting rule, then a second rate-limiting rule does not exist.
[0098] The current compute node executes SQL statements, which can be parsed and optimized according to the connection protocol to generate corresponding execution plans. The SQL execution engine then executes the plans. The current compute node summarizes the initial query results returned from each shard and performs subsequent calculations, such as aggregation, sorting, and filtering, to obtain the final query results, which are then returned to the target user.
[0099] In this embodiment, if the target user is not in the preset whitelist information or the target user is in the preset whitelist information but there is no corresponding rate limiting rule, it can accurately filter SQL statements that do not need to be rate limited, so as to provide a better user experience for the target user.
[0100] Figure 5 This diagram illustrates a flowchart of another distributed database rate limiting method provided in an embodiment of this disclosure. Figure 1 Based on the previous embodiment, S108 is added after S106 to limit the storage of SQL statements that satisfy the second rate limiting rule. For example... Figure 5 As shown, in one embodiment, a distributed database rate limiting method provided by this disclosure includes S102 to S108, wherein, after S106 if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record, and after executing the corresponding rate limiting operation, the method further includes:
[0101] S108. If the buffer is not full, SQL statements that meet the second rate limiting rule will be written to the buffer in batches; if the buffer is full or a timed flush task is performed, SQL statements that meet the second rate limiting rule will be written from the buffer to the database, thereby reducing the query of normal business SQL statements and facilitating subsequent business optimization.
[0102] To deepen the understanding of this publicly available distributed database rate limiting method, the following section, in conjunction with the appendix, provides further details. Figures 6-8 Please provide an explanation.
[0103] like Figure 6 As shown, the distributed database rate limiting method disclosed in this paper mainly includes the following steps:
[0104] S601. Obtain the target user's SQL statement;
[0105] S602, the SQL parser parses SQL statements;
[0106] S603. Determine the whitelist;
[0107] S604. When the target user is in the whitelist, execute normal SQL processing;
[0108] S605. When the target user is not in the whitelist, determine whether there is a rate limiting rule. The rate limiting rule includes the first rate limiting rule and the second rate limiting rule. If there is no rate limiting rule, execute normal SQL processing.
[0109] S606. When rate limiting rules exist, create a rate limiting detection module;
[0110] S607. When an SQL statement is already in the Watch record, execute the corresponding rate limiting operation in the Watch record.
[0111] S608. When the SQL statement is not in the Watch record, add a rate limiting detection module;
[0112] S609, Execute SQL processing with rate limiting detection.
[0113] It should be noted that the matching method between the target user and the first rate limiting rule, as well as the matching method between the SQL statement and the second rate limiting rule, can be referred to the aforementioned embodiments, and will not be repeated here.
[0114] like Figure 7 As shown, when the SQL execution coroutine triggers the rate limiting rule, the Watch record channel coroutine saves concurrently and stores the Watch record in the metadata table of the first shard of the current compute node, thus obtaining the Watch record table. Other compute nodes in the distributed database periodically pull the Watch records from the Watch record table to achieve synchronization of Watch records between different compute nodes, ensuring the consistency of information between different compute nodes.
[0115] like Figure 8 As shown, when the SQL execution coroutine triggers the rate limiting rule, the rate limiting SQL Channel writes the SQL statements that meet the second rate limiting rule into the buffer if the buffer is not full; when the buffer is full or a timed flush task is performed, the SQL statements that meet the second rate limiting rule are written from the buffer into the database. In order to reduce the query of normal business SQL, the SQL statements can be submitted in batches.
[0116] In this embodiment, the ability of a rate-limiting firewall based on SQL execution time can be achieved without introducing a centralized component. At the same time, the SQL statements that trigger rate limiting are stored to facilitate subsequent business optimization.
[0117] Based on the same inventive concept, this disclosure also provides a distributed database rate limiting device, as shown in the following embodiment. Since the principle by which this device solves the problem is similar to that of the method embodiment described above, the implementation of this device embodiment can refer to the implementation of the method embodiment described above, and repeated details will not be elaborated further.
[0118] Figure 9 This diagram illustrates the structure of a distributed database rate limiting device provided in an embodiment of this disclosure. Figure 9 As shown, the distributed database rate limiting device of this disclosure includes a statement acquisition module 910, a rule determination module 920, and a rate limiting execution module 930.
[0119] Among them, the statement acquisition module 910 is used to acquire the SQL statement of the target user;
[0120] The rule determination module 920 is used to determine the second rate limiting rule for the SQL statement based on the first rate limiting rule if the target user meets the first rate limiting rule.
[0121] The rate limiting execution module 930 is used to execute the corresponding rate limiting operation if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record.
[0122] It should be noted that the statement acquisition module 910, rule determination module 920, and rate limiting execution module 930 mentioned above correspond to S102 to S106 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.
[0123] In one embodiment, the rate limiting execution module 930 is used to execute SQL processing with rate limiting detection if the SQL statement satisfies the second rate limiting rule and the SQL statement is not in the monitoring record.
[0124] In one embodiment, the apparatus further includes a monitoring record generation module (not shown in the figures). The monitoring record generation module is used to generate a monitoring record for the SQL statement according to the rate limiting operation configured in the second rate limiting rule if the execution time of the SQL statement meets the preset SQL rate limiting conditions; and to store the monitoring record of the SQL statement in the target shard of the current computing node so that other computing nodes in the distributed database can pull the monitoring record of the SQL statement.
[0125] In one embodiment, a preset SQL rate limiting condition is configured in the second rate limiting rule. The preset SQL rate limiting condition includes a threshold for the execution time of the SQL statement that triggers rate limiting. The monitoring record generation module is used to determine that the execution time of the SQL statement meets the preset SQL rate limiting condition if the execution time of the SQL statement reaches the threshold for the execution time of the SQL statement that triggers rate limiting.
[0126] In one embodiment, the rate limiting rule includes at least one of the following: rate limiting rule name, rate limiting operation after triggering rate limiting, threshold for the duration of continuous effect after triggering rate limiting, priority of rate limiting rule, threshold for the execution time of the SQL statement that triggers rate limiting, and SQL matching pattern.
[0127] It should be noted that the first rate limiting rule includes at least one of the following: the keyword in the SQL statement that relates to the second rate limiting rule, the user to be matched for SQL rate limiting, and the user address to be matched for SQL rate limiting.
[0128] In one embodiment, the rate limiting execution module 30 is further configured to, after obtaining the target user's SQL statement, execute the SQL statement to obtain the query result if the target user exists in the preset whitelist information, or if the target user does not exist in the preset whitelist information and there is no first rate limiting rule and second rate limiting rule, and then send the query result to the target user.
[0129] In one embodiment, the device further includes a data storage module not shown in the figures. The data storage module is used to, if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record, execute the corresponding rate limiting operation, and if the buffer is not full, write the SQL statements that satisfy the second rate limiting rule into the buffer in batches; if the buffer is full or a timed flush task is performed, write the SQL statements that satisfy the second rate limiting rule from the buffer into the database.
[0130] In this embodiment, the current computing node obtains the SQL statement of the target user; if the target user meets the first rate limiting rule, the second rate limiting rule of the SQL statement is determined according to the first rate limiting rule; if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, the corresponding rate limiting operation is executed. This disclosure determines whether to execute the rate limiting operation based on the judgment result of whether the SQL statement meets the rate limiting rule and whether the SQL statement is already in the monitoring record, thereby improving the scalability and stability of the system without relying on centralized components.
[0131] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented in the following forms: entirely hardware implementations, entirely software implementations (including firmware, microcode, etc.), or implementations combining hardware and software aspects, collectively referred to herein as “circuits,” “modules,” or “systems.”
[0132] The following reference Figure 10 To describe an electronic device 1000 according to this embodiment of the present invention. Figure 10The electronic device 1000 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0133] like Figure 10 As shown, the electronic device 1000 is manifested in the form of a general-purpose computing device. The components of the electronic device 1000 may include, but are not limited to: at least one processing unit 1010, at least one storage unit 1020, and a bus 1030 connecting different system components (including storage unit 1020 and processing unit 1010).
[0134] The storage unit stores program code, which can be executed by the processing unit 1010 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 1010 can perform, as follows: Figure 1 The current computing node shown obtains the target user's SQL statement; if the target user meets the first rate limiting rule, then the second rate limiting rule for the SQL statement is determined according to the first rate limiting rule; if the SQL statement meets the second rate limiting rule and the SQL statement is already in the monitoring record, then the corresponding rate limiting operation is executed.
[0135] Storage unit 1020 may include readable media in the form of volatile storage units, such as random access memory (RAM) 10201 and / or cache memory 10202, and may further include read-only memory (ROM) 10203.
[0136] Storage unit 1020 may also include a program / utility 10204 having a set (at least one) program module 10205, such program module 10205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0137] Bus 1030 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the multiple bus structures.
[0138] Electronic device 1000 can also communicate with one or more external devices 1040 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with the system, and / or any device that enables the electronic device 1000 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1050. Furthermore, the system can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1060. Figure 10 As shown, network adapter 1060 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although... Figure 10 As not shown, other hardware and / or software modules may be used in conjunction with the electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0139] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0140] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, performs the steps described in the foregoing "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.
[0141] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0142] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.
[0143] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0144] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0145] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0146] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0147] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0148] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.
Claims
1. A distributed database rate limiting method, characterized in that, The method includes: Obtain the target user's Structured Query Language (SQL) statement; If the target user meets the first rate limiting rule, then the second rate limiting rule for the SQL statement is determined according to the first rate limiting rule. If the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record, then the corresponding rate limiting operation is executed.
2. The distributed database rate limiting method according to claim 1, characterized in that, The method further includes: If the SQL statement satisfies the second rate limiting rule and the SQL statement is not in the monitoring record, then SQL processing with rate limiting detection is executed.
3. The distributed database rate limiting method according to claim 1 or 2, characterized in that, The method further includes: If the execution time of the SQL statement meets the preset SQL rate limiting conditions, then a monitoring record of the SQL statement is generated according to the rate limiting operation configured in the second rate limiting rule; The monitoring records of the SQL statements are stored in the target shard of the current computing node, so that other computing nodes in the distributed database can pull the monitoring records of the SQL statements.
4. The distributed database rate limiting method according to claim 3, characterized in that, The preset SQL rate limiting condition is configured in the second rate limiting rule, and the preset SQL rate limiting condition includes a threshold for the execution time of the SQL statement that triggers rate limiting; The method further includes: If the execution time of the SQL statement reaches the threshold of the SQL statement execution time that triggers rate limiting, then the execution time of the SQL statement is determined to meet the preset SQL rate limiting conditions.
5. The distributed database rate limiting method according to claim 1, characterized in that, The second rate limiting rule includes at least one of the following: rate limiting rule name, rate limiting operation after triggering rate limiting, threshold for the duration of continuous effect after triggering rate limiting, priority of rate limiting rule, threshold for the execution time of the SQL statement that triggers rate limiting, and SQL matching pattern.
6. The distributed database rate limiting method according to claim 1, characterized in that, The first rate limiting rule includes at least one of the following: the keyword of the SQL statement that is associated with the second rate limiting rule, the user to be matched for SQL rate limiting, and the address of the user to be matched for SQL rate limiting.
7. The distributed database rate limiting method according to claim 1, characterized in that, After obtaining the target user's SQL statement, the method further includes: If the target user exists in the preset whitelist information, or if the target user does not exist in the preset whitelist information and neither the first rate limiting rule nor the second rate limiting rule exists, then the SQL statement is executed to obtain the query result, and the query result is sent to the target user.
8. The distributed database rate limiting method according to claim 1, characterized in that, After the step of executing the corresponding rate limiting operation if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record, the method further includes: If the buffer is not full, SQL statements that satisfy the second rate limiting rule will be written to the buffer in batches; If the buffer is full or a timed flush task is performed, an SQL statement that satisfies the second rate limiting rule will be written from the buffer to the database.
9. A distributed database rate limiting device, characterized in that, include: The statement retrieval module is used to retrieve the SQL statements of the target user; The rule determination module is used to determine a second rate limiting rule for the SQL statement based on the first rate limiting rule if the target user meets the first rate limiting rule. The rate limiting execution module is used to execute the corresponding rate limiting operation if the SQL statement satisfies the second rate limiting rule and the SQL statement is already in the monitoring record.
10. An electronic device, characterized in that, include: processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the distributed database rate limiting method as described in any one of claims 1-8 by executing the executable instructions.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed database rate limiting method as described in any one of claims 1-8.
12. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which are loaded and executed by a processor to enable the computer to implement the distributed database rate limiting method as described in any one of claims 1-8.