Request processing method and application gateway for distributed data cluster
By implementing syntax transformation, rate limiting rules, and caching mechanisms for Elasticsearch requests, we have addressed Elasticsearch's performance and security bottlenecks, achieving efficient traffic management and enhanced security, simplifying operational processes, and ensuring cluster stability and availability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- KANGCHENG INVESTMENT CHINA
- Filing Date
- 2026-01-12
- Publication Date
- 2026-06-05
AI Technical Summary
The Elasticsearch distributed search and analytics engine has bottlenecks in performance, security, and traffic control, resulting in high operational complexity, increased response latency, insufficient security, high version upgrade costs, and inadequate traffic control affecting stability.
This paper provides a request processing method and application gateway for distributed data clusters. It processes requests through syntax conversion and optimization, intercepts illegal operations, implements rate limiting rules, caches high-frequency query results, and realizes traffic replication and dual write, thereby improving cluster availability and security.
It improves the availability and security of Elasticsearch, reduces operational complexity, simplifies resource consumption and operational processes, and enables zero-disruption cluster version upgrades and migrations.
Smart Images
Figure CN122160433A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data, and in particular to a request processing method and application gateway for a distributed data cluster. Background Technology
[0002] With the widespread application of big data technology, Elasticsearch, as a mainstream distributed search and analytics engine, faces challenges such as performance bottlenecks, security risks, complex version upgrades, and insufficient traffic control. Existing architectures that directly use Elasticsearch have the following drawbacks: Performance issues: High-frequency queries and index operations can easily lead to excessive cluster load and increased response latency; Inadequate security: Lack of effective protection against malicious attacks (such as SQL injection and brute-force attacks); Poor version compatibility: Upgrading and migrating between different versions of Elasticsearch is costly and business continuity is difficult to guarantee; Lack of traffic control: Indiscriminate requests can easily cause cluster overload and affect stability.
[0003] Providing a request processing method and application gateway for distributed data clusters that can implement functions such as traffic control, request intervention, and query acceleration for Elasticsearch is of great significance for reducing operational complexity and improving operational efficiency. Summary of the Invention
[0004] The technical problem to be solved by this invention is to provide a request processing method and application gateway for a distributed data cluster, which realizes functions such as traffic control, request intervention, and query acceleration, significantly improving the availability, security and performance of Elasticsearch, thereby reducing the complexity of operation and maintenance.
[0005] To address the aforementioned issues, this invention provides a request processing method for a distributed data cluster, comprising: obtaining a request to be processed, and performing syntax conversion and optimization processing on the operation statements in the request to be processed, so that the request to be processed can be recognized by the target cluster and return the required fixed fields, wherein the operation of the request to be processed includes querying or writing to the target cluster; real-time detection of the request content of the request to be processed after syntax conversion and optimization processing, intercepting the request content containing illegal operations and recording it in the attack log; filtering out legitimate requests from the uninterrupted requests to be processed according to a preset rate limiting rule; parsing out the target index from the legitimate requests, and routing the legitimate requests to the node in the target cluster where the target index is located; caching the query results of high-frequency queries, and directly returning the cached query results when receiving duplicate queries; and performing traffic replication and dual writing on the legitimate requests according to preset routing rules when performing cluster version upgrades or cluster migrations.
[0006] In some embodiments, the step of obtaining the request to be processed further includes: parsing an intermediate format request according to the query format of the request sent by the requester; performing syntax parsing on the intermediate format request; and taking the intermediate format request with correct syntax as the request to be processed.
[0007] In some embodiments, the illegal operations include injecting queries and brute-force data retrieval.
[0008] In some embodiments, the rate limiting rules include at least one of the following: rate limiting based on upstream application, rate limiting based on data index, rate limiting based on cluster, and rate limiting based on requests per second.
[0009] In some embodiments, the rate limiting rules and the routing rules can be dynamically configured.
[0010] In some embodiments, the method further includes: triggering a circuit breaker when the target cluster is in a high-load or faulty state, thereby blocking all pending requests to the target cluster.
[0011] In some embodiments, the method further includes: when parsing the target index from the legitimate request fails, routing the legitimate request to a random node in the target cluster.
[0012] In some embodiments, when performing a cluster version upgrade or cluster migration, the step of performing traffic replication and dual writing on the legitimate request according to a preset routing rule further includes: distributing the legitimate request to a traffic replication node / cluster according to the traffic replication node / cluster in the routing rule; if the legitimate request is a write request, writing the service data of the legitimate request to the dual writing node / cluster according to the dual writing node / cluster in the routing rule.
[0013] In some embodiments, the request processing method for the distributed data cluster further includes logging and monitoring metrics, including at least one of CPU utilization, memory usage, interface QPS, request success rate, average response time, and slow query percentage.
[0014] To address the aforementioned issues, this invention also provides an application gateway for a distributed data cluster, comprising: a request intervention module, used to obtain requests to be processed and perform syntax conversion and optimization on the operation statements in the requests to be processed, enabling the requests to be processed to be recognized by the target cluster and return fixed required fields, wherein the operations of the requests to be processed include querying or writing to the target cluster; a traffic control module, used to perform real-time detection on the request content of the requests to be processed after processing by the request intervention module, intercept requests to be processed containing illegal operations and record them in the attack log, and filter out legitimate requests from the uninterrupted requests to be processed according to preset rate limiting rules; a query acceleration module, used to parse the target index from the legitimate requests, route the legitimate requests to the node in the target cluster where the target index is located, and cache the results of high-frequency queries so that the cached results are directly returned when duplicate queries are received; and a disaster recovery module, used to perform traffic replication and dual writing on the legitimate requests according to preset routing rules when performing cluster version upgrades or cluster migrations.
[0015] The above technical solution improves development efficiency by transforming and optimizing the operation statements in the requests to be processed; enhances the security of the data cluster by intercepting illegal operations; prevents cluster overload by setting pre-defined rate limiting rules and implementing multi-dimensional rate limiting; reduces internal forwarding overhead by accurately delivering requests to the corresponding nodes by obtaining the target index; improves request response speed by caching high-frequency query results and directly returning cached results when duplicate queries are received; and achieves zero-interruption cluster version upgrades and cluster migrations through traffic replication and dual writes. It not only improves the security and operational performance of the data cluster but also reduces resource consumption and simplifies operation and maintenance processes.
[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a request processing method for a distributed data cluster provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of an application gateway for a distributed data cluster provided in an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions in the embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] Figure 1 This is a flowchart of a request processing method for a distributed data cluster provided in an embodiment of the present invention.
[0021] This invention is geared towards Elasticsearch, and in this embodiment, the distributed data cluster is an Elasticsearch cluster.
[0022] like Figure 1 As shown, the request processing method of the distributed data cluster includes: Step S11, obtaining a request to be processed, and performing syntax conversion and optimization processing on the operation statements in the request to be processed, so that the request to be processed can be recognized by the target cluster and return the required fixed fields, wherein the operation of the request to be processed includes querying or writing to the target cluster; Step S12, detecting the request content of the request to be processed after syntax conversion and optimization processing in real time, intercepting the request content containing illegal operations and recording it in the attack log; Step S13, filtering out legitimate requests from the uninterrupted requests to be processed according to preset rate limiting rules; Step S14, parsing the target index from the legitimate requests and routing the legitimate requests to the node in the target cluster where the target index is located; Step S15, caching the results of high-frequency queries, and directly returning the cached results when receiving duplicate queries; Step S16, when performing cluster version upgrades or cluster migrations, performing traffic replication and dual writing on the legitimate requests according to preset routing rules.
[0023] Regarding step S11, a request to be processed is obtained, and the operation statements in the request to be processed are subjected to syntax conversion and optimization processing so that the request to be processed can be recognized by the target cluster and return the required fixed fields. The operation of the request to be processed includes querying or writing to the target cluster.
[0024] The target cluster is the data cluster involved in the operation of the request to be processed. The operation of the request to be processed is a query or a write operation on the target cluster.
[0025] The steps for obtaining the request to be processed specifically include: step S111, parsing the query format of the request sent by the requester to obtain an intermediate format request; step S112, performing syntax parsing on the intermediate format request, and taking the intermediate format request with correct syntax as the request to be processed.
[0026] In step S111, the query format includes the native Elasticsearch syntax format and the Druid SQL format. Requests sent by the requester are categorized according to the different query formats and parsed accordingly.
[0027] In step S112, Lucene syntax parsing is used. Elasticsearch relies on Lucene for retrieval, and Lucene syntax parsing and validation ensure that the request to be processed can be recognized by Elasticsearch.
[0028] This method provides a concise and easy-to-use encapsulation interface, which uses an Application Programming Interface (API) to encapsulate and simplify Elasticsearch's native syntax. Developers can use this encapsulation interface to simplify operation statements in requests and improve development efficiency.
[0029] In this step, the encapsulated interface is translated in reverse into native Elasticsearch syntax that can be recognized, such as Elasticsearch's native syntax, so that the target cluster can recognize it.
[0030] This step also includes: logging failed conversion requests and the reasons for failure in the log, setting the level to Error (ERROR), and logging each successful conversion request in the log, setting the level to Information (INFO).
[0031] In this embodiment, the target cluster is an Elasticsearch cluster. After performing syntax conversion on the operation statements in the request to be processed, the operation statements in the request to be processed have been converted into native Elasticsearch syntax. This step will also optimize the converted operation statements so that the target cluster returns the fixed required fields.
[0032] Specifically, by using Elasticsearch's native syntax for field filtering, the system returns fixed fields that are necessary for the business, reducing unnecessary data returns. This reduces network transmission and resource consumption (meeting the performance and traffic control requirements of the ES cluster), while also ensuring data security and simplifying business processing.
[0033] For example, in some implementations, the _source parameter is used to specify the returned fields; setting "size: 0" disables the return of the original document to avoid redundant data.
[0034] This step also includes logging the operation statements for optimizing the pending request and setting the level to INFO.
[0035] In some embodiments, this step can directly optimize the original operation statements in the request to be processed. Accordingly, there is no necessary order in which the syntax transformation and optimization processes are performed on the operation statements in the request to be processed.
[0036] Regarding step S12, the request content of the pending request after syntax conversion and optimization is detected in real time, and pending requests whose content contains illegal operations are intercepted and recorded in the attack log.
[0037] The illegal operations include SQL injection and brute-force data fetching. SQL injection refers to manipulating the database to perform operations beyond one's intended purpose by injecting malicious SQL statements into input fields or requests. Brute-force data fetching refers to disregarding system design rules and capacity, and forcibly acquiring data through high concurrency, large batches, and unrestrained methods.
[0038] This step inspects the request content of each received request after the syntax conversion and optimization process in step S11. Requests containing illegal operations are intercepted and recorded in the attack log. The attack log can be used to analyze system anomalies. In this embodiment, in addition to the request, the attack log also records its source, request time, etc.
[0039] In this embodiment, illegal operations can be detected by matching specific characters or specific regular expressions in the request content.
[0040] This step also includes logging uninterrupted pending requests and setting the log level to INFO.
[0041] Regarding step S13, legitimate requests are selected from the unblocked pending requests according to preset rate limiting rules. These rate limiting rules are used to set the conditions for traffic restriction.
[0042] The rate limiting rules include at least one of the following: rate limiting based on upstream applications, rate limiting based on data indexes, rate limiting based on clusters, and rate limiting based on requests per second (QPS).
[0043] Upstream application rate limiting refers to setting a traffic threshold for specific upstream applications. When the traffic sent by an upstream application exceeds or equals the application traffic threshold, its requests will be blocked. The upstream application is the sender of the request to be processed, distinguished by its IP address. In this embodiment, each upstream application is assigned its own independent application traffic threshold.
[0044] Data index rate limiting refers to setting a traffic threshold for a specific index within a specific data cluster. When the traffic to an index exceeds or exceeds the threshold, the corresponding request will be blocked. An index is similar to a "table". In this embodiment, each index in each data cluster is assigned its own independent traffic threshold.
[0045] Cluster rate limiting refers to setting a threshold for the traffic of a specific data cluster. When the traffic of a data cluster exceeds or equals the threshold, the corresponding requests will be blocked. In this embodiment, each data cluster is assigned its own independent cluster traffic threshold.
[0046] Application traffic thresholds, index traffic thresholds, and cluster traffic thresholds can be limited by the number of requests or by the amount of data in a single request.
[0047] QPS rate limiting refers to setting QPS traffic thresholds for different dimensions. For example, in this embodiment, QPS traffic thresholds are assigned to the upstream application dimension, the index dimension, and the cluster dimension respectively. When the QPS traffic of an upstream application is greater than or equal to the application QPS traffic threshold, its requests will be blocked; when the QPS traffic of an index is greater than or equal to the index QPS traffic threshold, its corresponding requests will be blocked; when the QPS traffic of a data cluster is greater than or equal to the cluster QPS traffic threshold, its corresponding requests will be blocked.
[0048] The various rules in the rate limiting rules work together to perform multi-dimensional rate limiting on the target cluster.
[0049] The speed limit rules can be configured via a configuration file or an interactive interface, and the configuration content is stored in a database.
[0050] The speed limiting rules can be dynamically configured. For example, if the speed limiting rules are configured via a configuration file, then when the configuration file corresponding to the speed limiting rules is updated, the speed limiting rules will be updated according to the new configuration file content; if the speed limiting rules are configured via an interactive interface, then when the configuration is successfully modified in the interactive interface, the speed limiting rules will be updated according to the new configuration content.
[0051] This step also includes: logging the filtered-out pending requests and their reasons, setting the level to ERROR, and logging the valid requests, setting the level to INFO.
[0052] Furthermore, when the target cluster is under high load or in a faulty state, a circuit breaker is triggered, blocking all pending requests to the target cluster. This circuit breaker ensures the stability of the target cluster.
[0053] In the case of circuit breaker failure, this step also includes logging the triggering of the circuit breaker and the reason for the circuit breaker failure, and setting the level to FATAL.
[0054] This step is used to prevent the target cluster from becoming overloaded. The pending requests that were not intercepted in step S12 and meet the rate limiting rules in step S13 are considered legitimate requests and proceed to the next step.
[0055] Regarding step S14, the target index is parsed from the legitimate request, and the legitimate request is routed to the node in the target cluster where the target index is located.
[0056] An index is a logical unit for storing data in Elasticsearch, and the target index is the index involved in the operation of the legitimate request. Specifically, the target index is parsed from the request content of the legitimate request, for example, extracted from request parameters or the URL.
[0057] Furthermore, this step also includes: obtaining the correspondence between indexes, shards, and nodes, and then, based on the target index, obtaining the node where the target index resides. In this embodiment, this can be obtained through Elasticsearch's API or library functions.
[0058] This step also includes logging pending requests that failed to be routed to the node containing the target index and the reasons for the failure, setting the level to ERROR, and logging pending requests that were successfully routed, setting the level to INFO.
[0059] When parsing the target index from a valid request fails, the valid request is routed to a random node in the target cluster. Correspondingly, the pending request that failed to parse the target index and the reason for the failure are logged with a warning (WARN) level, and the valid request that was routed to a random node in the target cluster is logged with an INFO level.
[0060] This step reduces the internal forwarding overhead of the target cluster by routing the legitimate request to the node where the target index is located in the target cluster, thus eliminating the need to receive the request from a non-target node in the target cluster and then forward it to the target node.
[0061] Regarding step S15, the results of high-frequency queries are cached, and the cached results are returned directly when a duplicate query is received.
[0062] The high-frequency query refers to the type of query that, within a unit of time (e.g., 1 second, 1 minute), has a significantly higher number of query requests for the same interface or the same data than other queries, and is continuously in a high-concurrency state, thus consuming a large proportion of system resources (CPU, memory, bandwidth, storage I / O).
[0063] Caching not only speeds up query results and improves response speed, but also reduces bandwidth consumption and saves hardware resources.
[0064] This step also includes: logging failed cached requests and the reasons for failure in the log, setting the level to ERROR, and logging successfully cached requests in the log, setting the level to INFO.
[0065] Regarding step S16, when performing a cluster version upgrade or cluster migration, traffic replication and dual writing are performed on the legitimate requests according to preset routing rules.
[0066] Specifically, the routing rules include traffic replication nodes / clusters and dual-write nodes / clusters. When the data cluster version is upgraded or the cluster is migrated, the legitimate requests, in addition to being routed to nodes in the target cluster, also include: (1) Distribute the legitimate request to the traffic replication node / cluster according to the traffic replication node / cluster in the routing rules. The legitimate request includes query request and write request.
[0067] (2) If the legitimate request is a write request, the business data of the legitimate request is written to both the target cluster and the dual-write node / cluster according to the dual-write node / cluster in the routing rules.
[0068] This step also includes: logging pending requests and reasons for failed traffic replication or double writes to the log, setting the level to ERROR, and logging pending requests for successful traffic replication or double writes to the log, setting the level to INFO.
[0069] This step can be used for canary deployments within a cluster. When an Elasticsearch cluster needs to undergo changes such as version upgrades or the deployment of new index structures, the canary cluster is configured in the routing rules described above. A small percentage (e.g., 10%) of the traffic from legitimate requests is copied to the canary cluster. After verifying the correctness of the canary cluster's results by comparing the return results of the canary cluster and the target cluster, the traffic percentage is gradually increased until a full switch to the canary cluster is achieved.
[0070] This step can be used for seamless cluster version upgrades. During the Elasticsearch cluster upgrade process, the standby cluster is configured in the routing rules described above. Through traffic replication, legitimate requests are simultaneously sent to both the target cluster and the standby cluster, ensuring that the standby cluster is synchronized with the latest data. During the upgrade, traffic routing temporarily switches all traffic to the standby cluster to perform the upgrade operation on the target cluster. After the upgrade, a small-scale traffic routing is used to verify the upgraded target cluster. Once confirmed to be correct, a full switchback is performed, and traffic replication is stopped. The upgrade process is seamless for users (service is uninterrupted, and requests are not lost), achieving a seamless experience.
[0071] This step can be used for disaster recovery. Configure the disaster recovery cluster in the routing rules, and through traffic replication, synchronously send all legitimate requests to both the target cluster and the disaster recovery cluster, ensuring real-time data consistency between the disaster recovery cluster and the target cluster. When a failure occurs and an anomaly is detected in the target cluster (such as downtime or response timeout), switch all traffic to the disaster recovery cluster. After the failure is recovered, switch back to the target cluster via routing, and simultaneously resume traffic replication. This allows services to quickly switch to the disaster recovery cluster in the event of a target cluster failure, avoiding business interruption and ensuring no data loss.
[0072] The routing rules can be configured via a configuration file or an interactive interface, and the configuration content is stored in a database.
[0073] The routing rules can be dynamically configured. For example, if the routing rules are configured via a configuration file, then when the configuration file corresponding to the routing rules is updated, the routing rules will be updated according to the new configuration file content; if the routing rules are configured via an interactive interface, then when the configuration is successfully modified in the interactive interface, the routing rules will be updated according to the new configuration content.
[0074] Correspondingly, the failure to configure the routing rule and the reason for the failure will be logged, and the level will be set to ERROR. The success of configuring the routing rule will be logged, and the level will be set to INFO.
[0075] The method for processing requests from the distributed data cluster further includes logging. The logging method in steps S11 to S16 is a specific embodiment of the present invention. Researchers can analyze the logs to troubleshoot faults and monitor the status of the data cluster.
[0076] The request processing method for the distributed data cluster also includes monitoring metrics. These metrics are used to monitor the cluster's status. For example, in some embodiments, the metrics include CPU utilization, memory usage, request success rate, average request response time, and slow query percentage.
[0077] By dynamically monitoring cluster status through logs and metrics, it can be used for slow query analysis and audit tracing.
[0078] This method improves development efficiency by transforming and optimizing the operation statements in the requests to be processed; enhances the security of the data cluster by intercepting illegal operations; prevents cluster overload by setting pre-defined rate limiting rules and implementing multi-dimensional rate limiting; reduces internal forwarding overhead by accurately delivering requests to the corresponding nodes by obtaining the target index; improves request response speed by caching high-frequency query results and directly returning cached results when duplicate queries are received; and achieves zero-interruption cluster version upgrades and cluster migrations through traffic replication and dual writes. These technical solutions not only improve the security and operational performance of the data cluster but also reduce resource consumption and simplify maintenance processes.
[0079] Based on the same inventive concept, this application also provides an application gateway for a distributed data cluster.
[0080] Figure 2 This is a schematic diagram of the structure of an application gateway for a distributed data cluster provided in an embodiment of the present invention.
[0081] like Figure 2 As shown, the application gateway 1 of the distributed data cluster includes: a request intervention module 11, a traffic control module 12, a query acceleration module 13, and a disaster recovery module 14.
[0082] The request intervention module 11 is used to obtain the request to be processed and perform syntax conversion and optimization processing on the operation statements in the request to be processed, so that the request to be processed can be recognized by the target cluster and return fixed required fields. The operation of the request to be processed includes querying or writing to the target cluster.
[0083] The traffic control module 12 is used to detect the request content of the pending requests after they have been processed by the request intervention module 11 in real time, intercept pending requests whose request content contains illegal operations and record them in the attack log, and filter out legitimate requests from the pending requests that have never been intercepted according to preset rate limiting rules.
[0084] The query acceleration module 13 is used to parse the target index from the legitimate request, route the legitimate request to the node in the target cluster where the target index is located, and cache the results of high-frequency queries so that the cached results are returned directly when a duplicate query is received.
[0085] The off-site disaster recovery module 14 is used to perform traffic replication and dual writing on the legitimate requests according to preset routing rules when performing cluster version upgrades or cluster migrations.
[0086] It should be noted that in at least one embodiment of this application, each module of the application gateway 1 of the distributed data cluster corresponds to each step of the aforementioned request processing method of the distributed data cluster. For the specific functions of the application gateway 1 of the distributed data cluster, please refer to the relevant description of the request processing method of the distributed data cluster, which will not be repeated here.
[0087] It should be noted that references to "an embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc., in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but each embodiment may not necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. In addition, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.
[0088] It should be noted that the terms "comprising" and "having," and their variations, used in this invention document are intended to cover non-exclusive inclusion. The terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence, unless explicitly indicated by the context. It should be understood that such data used interchangeably where appropriate. Furthermore, embodiments and features within embodiments of this invention can be combined with each other unless otherwise specified. In addition, descriptions of well-known components and technologies have been omitted in the above description to avoid unnecessarily obscuring the concepts of this invention. In the various embodiments described above, each embodiment focuses on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.
[0089] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A request processing method for a distributed data cluster, characterized in that, include: Obtain a request to be processed, and perform syntax transformation and optimization on the operation statements in the request to be processed, so that the request to be processed can be recognized by the target cluster and return the required fixed fields. The operation of the request to be processed includes querying or writing to the target cluster. The system detects the request content of the pending requests after syntax conversion and optimization in real time, intercepts pending requests whose content contains illegal operations, and records them in the attack log. According to the preset rate-limiting rules, legitimate requests are filtered out from the pending requests that have never been blocked; The target index is parsed from the legitimate request, and the legitimate request is routed to the node in the target cluster where the target index is located; Cache the results of frequently queried queries, and return the cached results directly when a duplicate query is received; When performing cluster version upgrades or cluster migrations, traffic replication and dual-write are performed on the legitimate requests according to preset routing rules.
2. The method according to claim 1, characterized in that, The step of obtaining the request to be processed further includes: The intermediate format request is obtained by parsing the query format of the request sent by the requester; The intermediate format request is parsed, and the intermediate format request that passes the syntax check is taken as a request to be processed.
3. The method according to claim 1, characterized in that, The illegal operations include injecting queries and brute-force data retrieval.
4. The method according to claim 1, characterized in that, The rate limiting rules include at least one of the following: rate limiting based on upstream applications, rate limiting based on data indexes, rate limiting based on clusters, and rate limiting based on requests per second.
5. The method according to claim 1, characterized in that, The rate limiting rules and the routing rules can be dynamically configured.
6. The method according to claim 1, characterized in that, The method further includes: triggering a circuit breaker when the target cluster is under high load or in a fault state, blocking all pending requests to the target cluster.
7. The method according to claim 1, characterized in that, The method further includes: when parsing the target index from the legitimate request fails, routing the legitimate request to a random node in the target cluster.
8. The method according to claim 1, characterized in that, When performing a cluster version upgrade or cluster migration, the steps of performing traffic replication and dual-write on the legitimate requests according to preset routing rules further include: According to the traffic replication nodes / clusters in the routing rules, the legitimate request is distributed to the traffic replication nodes / clusters; If the legitimate request is a write request, the business data of the legitimate request is written to the dual-write node / cluster according to the dual-write node / cluster in the routing rules.
9. The method according to claim 1, characterized in that, The method further includes logging and monitoring metrics, including at least one of CPU utilization, memory usage, interface QPS, request success rate, average response time, and slow query percentage.
10. An application gateway for a distributed data cluster, characterized in that, include: The request intervention module is used to obtain the request to be processed and perform syntax conversion and optimization processing on the operation statement in the request to be processed, so that the request to be processed can be recognized by the target cluster and return fixed required fields. The operation of the request to be processed includes querying or writing to the target cluster. The traffic control module is used to detect the request content of the pending requests after they have been processed by the request intervention module in real time, intercept pending requests whose request content contains illegal operations and record them in the attack log, and filter out legitimate requests from the pending requests that have never been intercepted according to preset rate limiting rules. The query acceleration module is used to parse the target index from the legitimate request, route the legitimate request to the node where the target index is located in the target cluster, and cache the results of high-frequency queries so that the cached results can be returned directly when a duplicate query is received. The off-site disaster recovery module is used to perform traffic replication and dual writing on the legitimate requests according to preset routing rules when performing cluster version upgrades or cluster migrations.