Gateway-based distributed data processing method, electronic equipment and program product

Abstract

The invention provides a distributed data processing method based on a gateway, electronic equipment and a program product. The method comprises the following steps: acquiring a data processing request sent by a service end; determining a request type and a routing rule corresponding to the request type according to the operation identifier analyzed from the data processing request; determining a target database cluster from a distributed database according to the request type and a corresponding routing rule; forwarding the data processing request to a target database cluster for execution; and feeding back an execution result of the target database cluster to the service end.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to gateway-based distributed data processing methods, electronic devices, and program products. Background Technology

[0002] In distributed search database applications, the requirements for high availability are increasing, especially in core business scenarios such as opportunity allocation, where they are widely used.

[0003] In existing technologies, commonly used dual-write solutions are highly invasive and fail to balance consistency and performance. Message queue dual-write requires business logic to be modified to write to the message queue, and the asynchronous mechanism inevitably introduces data latency; synchronous dual-write on the client side requires explicit calls to two sets of cluster APIs and cannot handle exceptions where one cluster succeeds while the other fails, resulting in high data consistency risks. An ideal dual-write solution needs to simultaneously meet the requirements of application transparency, latency sensitivity, and data consistency, but existing solutions struggle to achieve all three.

[0004] Common dual-read solutions often rely on simple heartbeat detection to achieve proximity-based reading or cold standby modes, failing to achieve stable load balancing and meet strict real-time requirements. The system can only judge based on the overall health of the cluster, unable to differentiate between the real-time needs of different requests. This can lead to high-real-time requests being routed to a standby cluster where data has not yet been synchronized, resulting in the inability to read the latest data. Therefore, a solution is needed that leverages a distributed database for data processing without the business being aware of it. Summary of the Invention

[0005] This disclosure provides a gateway-based distributed data processing method, electronic device, and program product.

[0006] According to a first aspect of this disclosure, a gateway-based distributed data processing method is provided. The method specifically includes: acquiring a data processing request sent by a business terminal; determining the request type and the corresponding routing rule based on an operation identifier parsed from the data processing request; wherein the request type includes read requests and write requests; determining a target database cluster from the distributed database based on the request type and the corresponding routing rule; forwarding the data processing request to the target database cluster for execution; and feeding back the execution result from the target database cluster to the business terminal.

[0007] According to the above solution, by introducing a gateway device, zero-intrusion access to the business end is achieved. The business end does not need to understand the complex dual-active architecture; it only needs to interact with a simple gateway interface to obtain highly available services, greatly reducing system transformation costs and technical barriers. Complex database resource scheduling, data read / write logic, and other processes are all encapsulated within the gateway device, enabling centralized management and operation of the system and improving the stability and maintainability of the overall architecture. Fine-grained intelligent routing (such as decisions based on real-time performance, session stickiness, and load balancing) effectively supports high-concurrency, low-latency business scenarios.

[0008] According to at least one embodiment of this disclosure, the request type includes: a write request; determining the target database cluster from the distributed database according to the request type and routing rules includes: if the request type is a write request, then determining the routing rule as a write request routing rule; determining the primary database cluster as the target database cluster from the distributed database according to the write request routing rule.

[0009] According to the above solution, by writing to the main database cluster in a unified manner, the data consistency problem in dual-write scenarios is effectively solved, ensuring that the distributed database is always in a consistent state. In this embodiment, the business side does not need to implement any dual-write or fault handling logic; all complexity is encapsulated in the gateway device, achieving true application transparency and zero-intrusion access.

[0010] According to at least one embodiment of this disclosure, a data processing request is forwarded to a target database cluster for execution; and the execution result of the target database cluster is fed back to the business end, including: routing a write request to the primary database cluster and writing the target data; feeding back the execution result of successfully writing the target data to the business end; and storing the target data in a message queue after combining it with a timestamp; and asynchronously replicating the target data to a backup database cluster in the order of timestamps through the message queue.

[0011] According to the above solution, a hybrid strategy of synchronous primary write and asynchronous backup replication ensures both a fast response experience for the business side and reliable data backup, perfectly balancing performance and reliability. Furthermore, the introduction of a timestamp mechanism resolves the operation order issue in an asynchronous replication environment, effectively avoiding the risk of data inconsistency that may arise from data restoration after fault recovery. The message queue, acting as a buffer layer, makes the data synchronization process between the primary and backup clusters completely transparent to the business side. Even if the backup cluster is temporarily unavailable, it will not affect the normal execution of the main process, greatly improving the overall availability of the system.

[0012] According to at least one embodiment of this disclosure, the request type includes: a read request; determining the target database cluster from the distributed database according to the request type and routing rules includes: if the request type is a read request, then determining the routing rule as a read request routing rule; determining the primary database cluster or the backup database cluster as the target database cluster from the distributed database according to the read request routing rule.

[0013] According to the above solution, by setting precise and intelligent read request routing rules, the technical problem that the dual-read solution cannot simultaneously satisfy real-time performance and load balancing can be effectively solved. Furthermore, this design is completely transparent to the business side; business applications do not need to be aware of the complex backend routing logic, achieving true application-unaware and zero-intrusion access.

[0014] According to at least one embodiment of this disclosure, determining the primary database cluster or backup database cluster as the target database cluster from the distributed database based on read request routing rules includes: determining whether the read request contains a real-time read identifier; wherein the real-time read identifier includes at least one of a custom real-time read identifier and a real-time read API; if there is a real-time read identifier, then determining the primary database cluster as the target database cluster from the distributed database.

[0015] According to the above solution, by identifying real-time read identifiers and forcibly routing data to the main database cluster, real-time data assurance for critical business scenarios is achieved. Unlike the nearest-neighbor read solution based on heartbeat detection, this embodiment achieves request-level real-time assurance, capable of distinguishing the real-time requirements of different requests.

[0016] According to at least one embodiment of this disclosure, determining the primary database cluster or the backup database cluster as the target database cluster from the distributed database based on read request routing rules includes: if the read request does not contain a real-time read identifier, parsing the read request to obtain a session ID; searching the session cluster mapping table in the gateway device based on the session ID; wherein, the session cluster mapping table is a table in the gateway device used to temporarily cache the mapping relationship between sessions and clusters; if the session ID is found in the session cluster mapping table, then the primary database cluster or the backup database cluster corresponding to the session ID in the session cluster mapping table is taken as the target database cluster.

[0017] According to the above solution, the session stickiness mechanism effectively prevents data inconsistency caused by the same session switching between primary and backup clusters. Furthermore, since this mechanism is implemented entirely within the gateway device, the business end does not need to be aware of the session ID processing, achieving truly zero-intrusion access and significantly reducing system modification costs.

[0018] According to at least one embodiment of this disclosure, determining the primary database cluster or the backup database cluster as the target database cluster from the distributed database based on read request routing rules includes: if a session ID is not found in the session cluster mapping table, obtaining the first load status information of the primary database cluster and the second load status information of the backup database cluster; performing load balancing calculation based on the first load status information and the second load status information; and determining the primary database cluster or the backup database cluster as the target database cluster based on the load balancing calculation result.

[0019] According to the above scheme, a real-time load balancing mechanism is used to achieve truly stable load sharing while ensuring the routing rules of the first two layers (real-time priority and session stickiness). Through fine-grained load status assessment and millisecond-level request-level decision-making, progressive and intelligent traffic distribution is achieved, maintaining the overall stable operation of the system even under uneven cluster load.

[0020] According to at least one embodiment of this disclosure, after determining the target database cluster from the distributed database, the method further includes: in response to an upgrade operation of the primary database cluster or the backup database cluster in the distributed database, determining whether the request version corresponding to the data processing request is the same as the database version of the target database cluster; if they are not the same, converting the protocol format of the data processing request through the protocol adaptation layer in the gateway device to obtain a compatible format data processing request, and sending the compatible format data processing request to the target database cluster.

[0021] The above solution achieves seamless business operation during rolling upgrades: during the upgrade observation period of several hours or even days, the business side does not need to be aware of the version differences in the backend cluster and can continue to use the new version API, while the gateway device automatically handles version compatibility issues to ensure business continuity. Furthermore, the business side does not need to be compatible with both the old and new APIs simultaneously, avoiding the burden of implementing complex version adaptation logic on the client side.

[0022] According to a second aspect of this disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs a first aspect of any embodiment of this disclosure.

[0023] According to a third aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a first aspect of any embodiment of this disclosure.

[0024] According to a fourth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a first aspect of any embodiment of this disclosure. Attached Figure Description

[0025] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0026] Figure 1 This is a schematic flowchart of a gateway-based distributed data processing method provided in an embodiment of the present disclosure.

[0027] Figure 2 This is a flowchart illustrating the method for determining a target database cluster for a read request, as provided in an embodiment of this disclosure.

[0028] Figure 3 This is a flowchart illustrating the write request execution process provided in an embodiment of this disclosure.

[0029] Figure 4 This is a flowchart illustrating the read request execution method provided in an embodiment of this disclosure.

[0030] Figure 5 This is a flowchart illustrating the method for determining the target database cluster in a read request provided in an embodiment of this disclosure.

[0031] Figure 6 This is a flowchart illustrating the method for requesting conversion when versions are inconsistent, as provided in embodiments of this disclosure.

[0032] Figure 7 This is a schematic diagram of the system architecture provided for an embodiment of this disclosure.

[0033] Figure 8 This is a schematic block diagram of a gateway-based distributed data processing device according to one embodiment of the present disclosure.

[0034] Figure 9 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure. Detailed Implementation

[0035] The present disclosure will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0036] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Figure 1This is a flowchart illustrating a gateway-based distributed data processing method provided in an embodiment of this disclosure. Figure 1 The method shown includes steps 101 to 104. This method can be executed by a gateway device.

[0038] Specifically, Figure 1 The method shown includes step 101: obtaining the data processing request sent by the business terminal.

[0039] In practical applications, the business side initiates data processing requests through its standard data access interface (such as the official SDK) without being aware that the backend is actually a complex dual-active architecture consisting of a primary database cluster and a backup database cluster.

[0040] It's important to note that when writing code, business developers only need to change the database connection address from pointing directly to an Elasticsearch cluster to pointing to the unified access address of the gateway device. Other than that, the API call logic, request format, and parameter settings on the business side do not require modification. When the business sends a data processing request (e.g., adding housing information or querying a list of neighborhoods), the communication process is the same as that of a regular database service. It is precisely because of the gateway device's existence and its handling of all the complex coordination work that the business can complete operations as if calling a regular API, achieving true application transparency and seamless interaction.

[0041] Step 102: Determine the request type and the corresponding routing rule based on the operation identifier parsed from the data processing request; the request type includes: read request and write request.

[0042] It should be noted that the operation identifiers mentioned here refer to the key characteristics that constitute an HTTP request, mainly including the HTTP verb (method) and the Uniform Resource Locator (URL) path. For example, when a request uses HTTP verbs such as PUT, POST, or DELETE, and its URL path points to a specific data document (such as / index / _doc / 123), the gateway device determines that the request is a write request; when a request uses the GET or POST verb but the URL path is / index / _search, it is determined to be a read request.

[0043] The routing rules mentioned here refer to the logical strategies pre-configured in the gateway device to guide request forwarding. Different request types have corresponding routing rules. For example, write requests correspond to the write request routing rule: synchronously write to the primary cluster to ensure strong consistency. Similarly, read requests correspond to the read request routing rule: dynamically select one of the primary and backup clusters as the target cluster based on consistency requirements and load conditions.

[0044] Step 103: Determine the target database cluster from the distributed database based on the request type and the corresponding routing rules.

[0045] It should be noted that the distributed database mentioned here refers to the overall data storage system composed of a primary database cluster and a backup database cluster. The target database cluster refers to the specific cluster instance (i.e., the primary database cluster or the backup database cluster) selected from the distributed database to handle the current request (e.g., a write request or a read request).

[0046] In practical applications, the gateway device selects a specific database cluster as the target database cluster from the distributed database containing the primary database cluster and the backup database cluster, based on the identified request type and the corresponding routing rules.

[0047] Step 104: Forward the data processing request to the target database cluster for execution; and feed back the execution results of the target database cluster to the business side.

[0048] Specifically, the gateway device establishes a communication connection with the target database cluster and forwards the data processing request. The target database cluster then performs the actual data read or write operation. After processing the request, the target database cluster generates an execution result (such as the dataset returned by the query or a confirmation message indicating successful write). The gateway device receives this result and returns it directly to the requesting service.

[0049] Based on the publicly available solutions described above, the introduction of gateway devices enables zero-intrusion access for business applications. Business applications do not need to understand complex active-active architectures; they only need to interact with a simple gateway interface to obtain highly available services, significantly reducing system modification costs and technical barriers. By encapsulating complex database resource scheduling, data read / write, and other logic within the gateway device, centralized management and maintenance of the system are achieved, improving the overall stability and maintainability of the architecture. Fine-grained intelligent routing (such as decisions based on real-time performance, session stickiness, and load balancing) effectively supports high-concurrency, low-latency business scenarios.

[0050] In one or more embodiments of this disclosure, such as Figure 2 This is a flowchart illustrating the method for determining a target database cluster for a read request, as provided in an embodiment of this disclosure. Figure 2 As shown, the request type in step 103 includes: write request; determining the target database cluster from the distributed database according to the request type and routing rules includes: Step 201: If the request type is a write request, then determine the routing rule as a write request routing rule. Step 202: Determine the primary database cluster as the target database cluster from the distributed database according to the write request routing rule.

[0051] In practical applications, when a gateway device receives a data processing request from a service and determines that it is a write request, it will execute a specific processing flow. A write request refers to a request that includes data addition, modification, or deletion operations, such as HTTP verbs like PUT, POST, or DELETE, or a URL path pointing to a specific data document, such as / properties / _doc / 12345.

[0052] When the gateway device recognizes a write request, it will automatically determine the applicable routing rule as a write request routing rule. The write request routing rule is a preset strategy specifically designed for handling write operations, ensuring consistency and reliability between the primary and backup databases during data writing. Based on this write request routing rule, the gateway device will determine the primary database cluster from the distributed databases as the target database cluster for processing the write request.

[0053] In a distributed environment, allowing simultaneous writes to multiple database clusters inevitably leads to the problem of "non-atomic double write operations." This raises the issue of situations where a write operation succeeds in cluster A but fails in cluster B, resulting in data inconsistency that is difficult to recover from automatically. Therefore, this routing rule fundamentally avoids this risk by forcing all write requests to be routed only to a single primary database cluster, ensuring the atomicity of data operations and the consistency of the system state.

[0054] Furthermore, the design of using the master database cluster as the "sole source of truth" effectively prevents multiple nodes from simultaneously believing themselves to be the master node and accepting write operations, which could lead to data conflicts and system chaos. This solution, through a centralized write control mechanism, ensures that all write operations are based on the same data state, thereby maintaining the integrity and reliability of the system.

[0055] When the gateway device resolves an operation identifier indicating a write request, it directly selects the primary database cluster as the target database cluster without considering cluster load status, network latency, or other complex factors. This unconditional routing decision ensures the consistency and predictability of all write request processing logic, avoiding behavioral differences caused by environmental changes.

[0056] Based on the publicly available solutions described above, by uniformly writing to the main database cluster, the data consistency problem in dual-write scenarios is effectively solved, ensuring that the distributed database is always in a consistent state. In this embodiment, the business side does not need to implement any dual-write or fault handling logic; all complexity is encapsulated in the gateway device, achieving true application transparency and zero-intrusion access.

[0057] In one or more embodiments of this disclosure, such as Figure 3This is a flowchart illustrating the write request execution process provided in an embodiment of this disclosure. Figure 3 As shown, step 104, which involves forwarding the data processing request to the target database cluster for execution and feeding back the execution result from the target database cluster to the business end, includes: Step 301: Routing the write request to the primary database cluster and writing the target data. Step 302: Feeding back the execution result of successfully writing the target data to the business end. And, Step 303: Combining the target data with a timestamp and storing it in a message queue. Step 304: Asynchronously replicating the target data to the backup database cluster in timestamp order via the message queue.

[0058] In practical applications, the gateway device establishes a communication connection with the main database cluster and forwards write requests sent by the application client directly or after version format conversion to the main database cluster. Upon receiving the request, the main database cluster executes the corresponding data processing logic, such as adding, modifying, or deleting specific documents.

[0059] Once the primary database cluster confirms a successful write, it immediately returns a response to the application, without waiting for the write result from the backup database cluster. This significantly reduces the write latency perceived by the application, ensuring rapid response capabilities in high-concurrency scenarios (such as 100,000+ requests per second) and meeting the application's requirements for low-latency response times.

[0060] Meanwhile, to ensure high availability and recovery capabilities, the gateway device stores the target data and timestamps together in a message queue. The timestamp is a strictly incrementing sequence number used to identify the relative order in which operations occur. By combining the target data (i.e., the specific content of the write operation) with this timestamp, a command packet containing complete operation information is formed and stored in the efficient and reliable message queue. The message queue, as an intermediate buffer mechanism, effectively decouples the direct dependencies between the primary and backup clusters, preventing the execution of the main process from being affected by the temporary unavailability of the backup cluster. Furthermore, it monitors the update status in the message queue and processes them strictly according to the order of the timestamps. When a command packet is retrieved from the message queue, the target data and timestamp information are parsed, and then the corresponding interface of the backup database cluster is called to execute the data write operation. Because the command packet contains strictly ordered timestamps, even in cases of network instability or temporary unavailability of the backup cluster, it ensures that all operations are ultimately executed on the backup database cluster in the correct order, thereby guaranteeing eventual data consistency between the primary and backup clusters.

[0061] Based on the publicly available solutions described above, the hybrid strategy of synchronous primary write and asynchronous backup replication ensures both a fast response experience for the business side and reliable data backup, perfectly balancing performance and reliability. Furthermore, the introduction of a timestamp mechanism resolves the operation order issue in an asynchronous replication environment, effectively avoiding the risk of data inconsistency that may arise from data restoration after fault recovery. The message queue, acting as a buffer layer, makes the data synchronization process between the primary and backup clusters completely transparent to the business side. Even if the backup cluster is temporarily unavailable, it will not affect the normal execution of the main process, greatly improving the overall availability of the system.

[0062] In one or more embodiments of this disclosure, such as Figure 4 This is a flowchart illustrating the read request execution method provided in an embodiment of this disclosure. Figure 4 As shown, the request type in step 103 includes: read request; determining the target database cluster from the distributed database according to the request type and routing rules includes: Step 401: If the request type is a read request, then determine the routing rule as a read request routing rule. Step 402: Determine the primary database cluster or backup database cluster as the target database cluster from the distributed database according to the read request routing rule.

[0063] It should be noted that the read request routing rules mentioned here refer to a dynamic decision-making mechanism specifically for read requests. Based on consistency requirements (such as data real-time requirements) and load conditions (such as the current resource consumption of the cluster), the target database cluster is selected after weighing the options between the primary database cluster and the backup database cluster.

[0064] Specifically, the read request rules evaluate the characteristics of read requests: if consistency requirements are high (e.g., needing to read the latest data), the primary database cluster is prioritized as the target database cluster to ensure strong consistency; if consistency requirements are low and load pressure is high, the backup database cluster is selected as the target database cluster to share the pressure on the primary database cluster and achieve high availability. The execution of this rule is a fine-grained three-layer intelligent routing for each read request individually (details will be explained in subsequent embodiments), rather than a coarse-grained judgment of the entire cluster. This achieves a shift from distributed cluster-level switching to each read request, from coarse-grained heartbeat detection to fine-grained multi-dimensional evaluation, and from passive database switching to active scheduling of routing rules. This fundamentally solves the technical problem that heartbeat-based proximity read / cold standby solutions cannot simultaneously achieve stable load balancing and real-time read requirements, such as avoiding traffic avalanches or backup cluster data lag caused by misjudgments of heartbeats.

[0065] The read scheduling module inside the gateway device collects the characteristics of read requests (such as request headers, API type, session ID) and real-time cluster metrics (such as CPU utilization, number of connections), and then applies read request routing rules to perform trade-off calculations. For example, if the read request involves high real-time requirements (such as index / get API), the primary database cluster is selected as the target database cluster after a trade-off; if it is a normal query and the primary database cluster is under high load, the backup database cluster is selected as the target database cluster.

[0066] Based on the publicly available solutions described above, it is evident that by setting precise and intelligent read request routing rules, the technical challenge of dual-read solutions simultaneously satisfying real-time performance and load balancing can be effectively addressed. Furthermore, this design is completely transparent to the business side; business applications do not need to be aware of the complex backend routing logic, achieving truly application-agnostic and zero-intrusion integration.

[0067] In one or more embodiments of this disclosure, such as Figure 5 This is a flowchart illustrating the method for determining the target database cluster in a read request provided in an embodiment of this disclosure. Figure 5 As shown, step 402, which involves determining the primary or backup database cluster as the target database cluster from the distributed database based on the read request routing rules, includes: Step 501: Determining whether the read request contains a real-time read identifier; wherein, the real-time read identifier includes at least one of a custom real-time read identifier and a real-time read API. Step 502: If a real-time read identifier exists, then the primary database cluster is determined as the target database cluster from the distributed database.

[0068] In practical applications, when a gateway device determines that a request is a read request and applies read request routing rules, it first performs a real-time requirement assessment on the read request. The gateway device analyzes whether the read request contains a real-time read identifier, which includes at least one of a custom real-time read identifier and a real-time read API.

[0069] It's important to note that a custom real-time read flag refers to a special marker that the business side can actively add when sending a request, such as `X-Real-Time: true` in the request header. When business developers know that a query must retrieve the latest data (e.g., a user immediately viewing order details after submitting an order), they can add this flag to the request, issuing a high real-time requirement instruction to the gateway device. Real-time read APIs refer to specific API interfaces that have high real-time requirements by default, such as `GET / _doc / {id}` or `GET / _source / {id}` for retrieving details of a single record. Based on its understanding of the business scenario, the gateway device treats such point-to-point query operations as high real-time requests by default, because users typically expect to see the data they just processed immediately.

[0070] When the gateway device detects that a read request contains any of the aforementioned real-time read identifiers, it will immediately determine the primary database cluster as the target database cluster from the distributed database. The reason for using the primary database cluster as the target cluster is that for queries with high real-time requirements, retrieving potentially outdated data from the backup database cluster could lead to user confusion or even business problems. For example, if a real estate agent has just updated property information but cannot see their changes on the details page, it will severely impact business trust.

[0071] As can be seen from the publicly available solutions described above, by identifying real-time read identifiers and forcibly routing data to the main database cluster, real-time data assurance for critical business scenarios is achieved. Unlike the nearest-neighbor read solution based on heartbeat detection, this embodiment achieves request-level real-time assurance, capable of distinguishing the real-time requirements of different requests.

[0072] For example, in the opportunity allocation business, for operations that require extremely high data freshness, such as updating agent information and changing property status, users can immediately see the latest results, significantly improving the business experience and system credibility. At the same time, this mechanism is completely transparent to the business side. Business applications only need to add identifiers or use specific APIs as needed, without needing to be aware of the complex routing logic in the backend, achieving true application-independent operation.

[0073] In one or more embodiments of this disclosure, such as Figure 5 As shown, determining the primary or backup database cluster as the target database cluster from the distributed database according to the read request routing rules includes: Step 503: If the read request does not contain a real-time read identifier, parse the read request to obtain the session ID. Step 504: Look up the session cluster mapping table in the gateway device based on the session ID; wherein, the session cluster mapping table is a table in the gateway device used to temporarily cache the mapping relationship between sessions and clusters. Step 505: If the session ID is found in the session cluster mapping table, then the primary or backup database cluster corresponding to the session ID in the session cluster mapping table is taken as the target database cluster.

[0074] In practical applications, when the gateway device determines that the read request does not contain a real-time read identifier, it will perform a second-layer routing decision: using the session cluster mapping table for session stickiness analysis. The gateway device first parses the read request to obtain the session ID, which is a unique identifier carried by the service client in the request to identify the user's session state. The session cluster mapping table is a table in the gateway device used to store the mapping relationship between sessions and clusters. It records the database cluster information most recently accessed by each session ID. This mapping table uses in-memory storage to ensure query efficiency and sets an appropriate time-to-live (TTL) to automatically clean up expired sessions. The TTL value can be flexibly configured according to business characteristics, typically ranging from a few seconds to a few minutes.

[0075] If the session ID is successfully found in the session cluster mapping table, the gateway device will directly use the primary or backup database cluster recorded in the mapping table as the target database cluster for the request. This ensures a consistent experience for the same user during continuous operations. For example, when a user browses multiple pages of a property listing, if the previous request is routed to the primary database cluster, subsequent related requests will also be routed to the same cluster, avoiding inconsistencies in page display due to data synchronization delays between the primary and backup clusters (e.g., the previous page shows available properties, but the next page shows no properties).

[0076] Based on the publicly available solutions described above, the session stickiness mechanism effectively prevents data inconsistency issues caused by the same session switching between primary and backup clusters. Furthermore, since this mechanism is implemented entirely within the gateway device, the business end does not need to be aware of the session ID processing, achieving truly zero-intrusion access and significantly reducing system modification costs.

[0077] For example, when agents view multiple property details or perform multiple steps, the system can maintain a consistent view of the data, significantly improving user experience and business reliability.

[0078] In one or more embodiments of this disclosure, such as Figure 5 As shown, determining the primary or backup database cluster as the target database cluster from the distributed database according to the read request routing rules includes: Step 506: If the session ID is not found in the session cluster mapping table, obtain the first load status information of the primary database cluster and the second load status information of the backup database cluster. Step 507: Perform load balancing calculation based on the first and second load status information, and determine the primary or backup database cluster as the target database cluster based on the load balancing calculation result.

[0079] In practical applications, when the gateway device cannot find the session ID in the session cluster mapping table, it will perform a Layer 3 routing decision: load balancing decision. The gateway device first obtains the first load status information of the primary database cluster and the second load status information of the backup database cluster. It should be noted that the load status information mentioned here includes, but is not limited to, key performance indicators such as CPU utilization, memory usage, network latency, current connection count, and query queue length. These indicators are collected and updated at millisecond-level frequency by a real-time monitoring system.

[0080] In practical implementation, mature strategies such as weighted round-robin or least-connections algorithms can be used. For example, in the weighted round-robin algorithm, the gateway device dynamically adjusts the weights based on the real-time load of each cluster, with clusters with lower loads receiving higher weights. In the least-connections algorithm, the gateway device selects the cluster with the fewest current connections as the target. This calculation process is completed within milliseconds, ensuring the real-time nature and accuracy of routing decisions.

[0081] Furthermore, based on the load balancing calculation results, the gateway device ultimately determines either the primary or backup database cluster as the target database cluster for the request. It's worth noting that this decision applies to read requests that have neither high real-time requirements nor session stickiness. For such requests, the system prioritizes resource utilization efficiency, routing the request to the cluster with the lighter current load, thereby achieving balanced utilization of resources across the two clusters.

[0082] Based on the publicly available solutions described above, a real-time load balancing mechanism achieves truly stable load sharing while ensuring the implementation of the first two layers of routing rules (real-time priority and session stickiness). Through fine-grained load status assessment and millisecond-level request-level decision-making, progressive and intelligent traffic allocation is achieved, maintaining the overall stable operation of the system even under uneven cluster load.

[0083] For example, when faced with high-concurrency query pressure of over 100,000 queries per second, the system can intelligently distribute traffic across two clusters, avoiding single-point overload and significantly improving system throughput and stability. Especially during peak business periods, this mechanism can automatically identify and avoid hotspot clusters. Furthermore, since load balancing decisions are entirely made autonomously by the gateway device, the business side does not need to be aware of the underlying resource allocation logic, achieving true application transparency and zero-intrusion access.

[0084] In one or more embodiments of this disclosure, such as Figure 6 This is a flowchart illustrating a method for requesting conversion when versions are inconsistent, as provided in embodiments of this disclosure. Figure 6 As shown, after determining the target database cluster from the distributed database, the process further includes: Step 601: In response to an upgrade operation of the primary or backup database cluster in the distributed database, determine whether the version of the data processing request is the same as the database version of the target database cluster. Step 602: If they are not the same, convert the protocol format of the data processing request through the protocol adaptation layer in the gateway device to obtain a compatible format data processing request, and then send the compatible format data processing request to the target database cluster.

[0085] In practical applications, after the gateway device determines the target database cluster from the distributed database, it also needs to handle version compatibility issues during rolling upgrades of the distributed database cluster. Because distributed databases cannot be terminated or suspended due to upgrades, they may be upgraded in batches. During the upgrade process, the primary and backup database clusters may be running different versions of software, leading to protocol incompatibility.

[0086] Specifically, when the primary or backup database cluster in a distributed database performs an upgrade operation, the gateway device executes the following steps: First, the gateway device determines whether the version of the data processing request matches the database version of the target database cluster. During this process, the gateway device analyzes the API version (i.e., the request version) used in the data processing request sent by the application and compares it with the actual version (i.e., the database version) of the target database cluster to which it will be forwarded. For example, if the application uses Elasticsearch 8.0 API features (such as Point-in-Time queries) to send a read request, while the target database cluster is still running version 7.10, the gateway device can identify the version mismatch.

[0087] Furthermore, the protocol format of data processing requests is converted by the protocol adaptation layer in the gateway device. This protocol adaptation layer is a specially designed component within the gateway device to resolve protocol compatibility issues between different database versions. Requests converted by the protocol adaptation layer are fully compatible with the version of the target database cluster, can be processed normally, and return the expected results. When a version mismatch is detected, the protocol adaptation layer initiates a conversion process: For requests, it converts the higher-version API to an equivalent API supported by the lower version. For responses, it converts the lower-version response back to a format compatible with the higher version.

[0088] For example, when the business uses the Point in Time (PIT) API of ES 8.0 to send a read request, while the target database cluster is running ES 7.10 (which does not support the PIT API), the protocol adaptation layer will convert the PIT API to the Scroll API supported by ES 7.10; after the target database cluster returns the Scroll result, the protocol adaptation layer will convert the result back to PIT format and finally feed it back to the business.

[0089] Based on the publicly available solutions described above, seamless business operations are achieved during rolling upgrades: During the upgrade observation period, which can last for hours or even days, the business side does not need to be aware of the version differences in the backend cluster and can continue using the new version API. The gateway device automatically handles version compatibility issues, ensuring business continuity. Furthermore, the business side does not need to be compatible with both the old and new APIs simultaneously, avoiding the burden of implementing complex version adaptation logic on the client side.

[0090] In one or more embodiments of this disclosure, the method further includes: periodically comparing the data differences between the primary database cluster and the backup database cluster; if differences exist, retrieving missing data from the primary database cluster and automatically repairing the backup database cluster to achieve end-to-end data consistency.

[0091] In practical applications, to further ensure the data reliability of distributed database systems, gateway devices also implement an end-to-end data consistency guarantee mechanism. This mechanism, as a powerful supplement to the aforementioned routing and replication processes, is specifically designed to detect and repair potential data discrepancies between the primary and backup database clusters.

[0092] In one alternative approach, the mechanism includes the following steps: periodically comparing data differences between the primary and backup database clusters. During this process, the system initiates a separate gateway-based distributed data processing service, which performs data consistency checks at preset time intervals (e.g., every 5 minutes).

[0093] Gateway-based distributed data processing services compare the data status of two clusters by: performing digest calculations on key data tables or indexes in the primary and backup database clusters, such as generating hash values ​​or checksums based on document ID ranges; comparing the digest values ​​of the same data range in the two clusters to identify potentially differing areas; and performing finer-grained comparisons on the identified areas of difference to pinpoint specific documents or records. This hierarchical comparison strategy avoids the performance overhead of full data scanning, ensuring that the impact of gateway-based distributed data processing on the normal operation of the system is minimized.

[0094] If discrepancies exist, the missing data is retrieved from the primary database cluster. Once the gateway-based distributed data processing service confirms the data discrepancies, it initiates a data repair process: generating a discrepancy report that clearly identifies missing or inconsistent data items in the backup database cluster; extracting the complete content of these missing data items from the primary database cluster; and packaging the extracted data into repair instructions, ready to send them to the backup database cluster. It is worth noting that this process considers the timeliness and importance of the data, prioritizing the repair of critical business data to ensure that core functions are not affected.

[0095] The system automatically repairs the standby database cluster to achieve end-to-end data consistency. It sends repair commands to the standby database cluster and executes data update operations. The end-to-end data consistency mentioned earlier refers to the consistency of all data replicas throughout the entire data writing process, from the source (business end) to the data storage endpoint (standby database cluster). This includes not only the consistency of data content but also the consistency of the data operation sequence, ensuring that the standby database cluster accurately reflects the state of the primary database cluster.

[0096] Based on the aforementioned publicly available solutions, the problem of implicit data inconsistency is resolved: In actual operation, due to network jitter, partial write failures, and other reasons, there may be subtle data differences between the primary and backup clusters. This mechanism can proactively detect and repair these differences, ensuring data integrity. Secondly, it achieves unattended automatic repair: Without manual intervention, the system can automatically complete difference detection, data retrieval, and repair operations, significantly reducing operational complexity and labor costs.

[0097] In one or more embodiments of this disclosure, the method further includes: assigning a unique tracking identifier to a data processing request; using the tracking identifier to achieve cross-cluster request tracing and fault location; and recording routing logs for monitoring and analysis.

[0098] In practical applications, to further improve system observability and operational efficiency, gateway devices also implement a full-link request tracing mechanism. This mechanism serves as an important supplement to the aforementioned routing, replication, and gateway-based distributed data processing flows, specifically designed for end-to-end request monitoring and rapid fault location.

[0099] Specifically, the mechanism includes the following key steps: First, a unique trace identifier is assigned to each data processing request. When the gateway device receives a data processing request from the service provider, it immediately generates a globally unique trace identifier for that request. This trace identifier is a string composed of a timestamp, a gateway device identifier, and a sequence number, ensuring that it will not be repeated during the system's runtime. For example, a typical trace identifier might look like TRACE-20231127-1510-0001, which includes date, timestamp, and sequence number information. This trace identifier flows throughout the system with the request, traversing the entire processing chain from the gateway device to the target database cluster.

[0100] Cross-cluster request tracing and fault location are achieved through tracking identifiers. In a distributed environment, a request may involve multiple system components and network jumps, making it difficult for traditional log analysis methods to fully reconstruct the request path. In this embodiment, the gateway device injects tracking identifiers into requests forwarded to the target database cluster, enabling the target database cluster to record the identifier in its logs. When system anomalies or performance issues occur, operations and maintenance personnel can quickly retrieve all records related to the request from the logs of each component based on the tracking identifier, fully reconstructing the request's processing path, time consumption, and status. This cross-cluster request tracing capability is particularly important for active-active architectures because it clearly shows whether the request is processed in the primary or backup database cluster, and what bottlenecks may be encountered during the processing.

[0101] The gateway device meticulously records the routing decision process for each request, including but not limited to: request type, real-time assessment result, session ID, reason for selecting the target database cluster, and request processing time. These routing logs not only contain basic access records but also rich contextual information, providing valuable data support for system monitoring and performance analysis. These logs are sent in real-time to a centralized log analysis platform to generate various monitoring metrics and visualization reports, such as cluster load distribution, request latency distribution, and error rate trends.

[0102] Based on the publicly available solutions described above, in complex active-active architectures, when performance degradation or service anomalies occur, operations and maintenance personnel can pinpoint the root cause of the problem within minutes by tracing identifiers, without having to manually sift through massive amounts of logs, thus shortening fault recovery time. Furthermore, it enables refined system monitoring: by analyzing routing logs, the actual effectiveness of the Layer 3 intelligent routing strategy can be accurately evaluated, such as the proportion of real-time requests, the success rate of session stickiness, and the uniformity of load balancing, providing data support for system optimization.

[0103] like Figure 7 This is a schematic diagram of the system architecture provided for an embodiment of this disclosure. Figure 7 As shown, the business terminal 10 accesses the backend distributed database 30 through the gateway device 20. This distributed database 30 includes a primary database cluster 31 and a backup database cluster 32. Both the primary database cluster 31 and the backup database cluster 32 are distributed search databases 30 (such as search clusters built based on Elasticsearch or similar technologies). They can be deployed in different availability zones or data centers to achieve a dual-active architecture. Alternatively, they can be deployed in the same region or data center.

[0104] The gateway device 20 has built-in functional modules for executing routing rules, such as write scheduling, read scheduling, and protocol adaptation. It is deployed between the business end 10 and the distributed database 30 and is responsible for receiving all data processing requests (read requests and write requests) sent by the business end.

[0105] In a typical implementation, the service terminal 10 sends a data processing request to the gateway device 20 via the HTTP / HTTPS protocol. The data processing request includes at least an HTTP verb and a URL path, and the gateway device 20 determines the request type by parsing this information. Then, based on the request type, it selects the corresponding routing rule and finds a suitable target database cluster from the distributed database 30 to execute the request task.

[0106] The following will use specific examples to illustrate... Figure 7 The specific process of implementing write and read requests in the system is explained in detail.

[0107] The data writing process is as follows: When business terminal 10 needs to add, modify, or delete data: Upon receiving a write request instruction, business terminal 10 sends a write data request to gateway device 20 via the standard SDK, as usual. Business terminal 10 is completely unaware of the existence of a distributed database 30 in the backend, believing it is only communicating with a single database, demonstrating true application transparency.

[0108] The data is synchronously written to the primary database cluster 31 (ensuring strong consistency). The write scheduling module in gateway device 20 immediately and synchronously writes this data to the primary database cluster 31. This method is used to ensure that the primary database cluster 31 is the sole source of truth for the data. As soon as the write to the primary database cluster 31 is successful, gateway device 20 immediately returns a write success response to the business terminal 10, ensuring both strong data consistency and meeting the business's stringent low latency requirements (within 200ms).

[0109] Asynchronous replication is performed to the backup database cluster 32 (ensuring high performance). While synchronously writing to the primary database cluster 31, the write scheduling module combines the write operation instruction (e.g., updating specific data at a specific location) with a strictly incrementing timestamp, packages it into a task, and stores it in a high-efficiency message queue. The message queue acts as an intermediate buffer layer, effectively decoupling the direct dependency between the primary and backup database clusters 30, allowing the primary database cluster 31 to operate without waiting for a response from the backup database cluster 32, thus significantly improving write performance. Subsequently, a dedicated data replication service retrieves these tasks from the message queue in an orderly manner.

[0110] Ordered replay (ensuring data order is not disrupted). Before executing each task, the data replication service processes them strictly in sequence according to timestamps. When receiving and executing these tasks, the standby database cluster 32 must execute them one by one in the order of timestamps. This design solves the core problem in asynchronous replication environments—ensuring the order of operations. For example, it ensures that the sequence of operations—deleting first and then adding—is not reversed, avoiding data logic chaos and achieving eventual consistency between the primary and standby database clusters 30.

[0111] Read data (ensuring high concurrency and load balancing). When business device 10 needs to query data, it sends a read request. That is, business device 10 sends a data query request to gateway device 20.

[0112] Request parsing. The read scheduling module parses the raw read request and extracts key features: Request type identifier: checks whether the request header contains explicit real-time flags such as X-Real-Time: true. Session state: extracts the SessionID information carried in the request.

[0113] Real-time request assessment: The read scheduling module performs a real-time requirement assessment. Assessment criteria include: the request header explicitly requires strong consistency (e.g., Consistency-Level: strong); the request uses a default API with high real-time performance (e.g., / index / get); and it involves a subsequent query that has just completed a write operation. If any of these conditions are met, the request is directly routed to the main database cluster 31, ensuring that the business terminal 10 can obtain the latest and most accurate data.

[0114] Stateful request assessment. The read scheduling module performs session stickiness analysis. Assessment criteria include: the request contains a valid SessionID or is a consecutive request within the same business context. Actions include: querying the session-cluster mapping table maintained internally by the gateway device 20 to locate the database cluster last accessed by the session ID; checking the health status of the cluster (via a real-time heartbeat detection mechanism); and if the cluster is healthy, routing the request to the same cluster to ensure data visibility during continuous user operations. Figure 1 To the point of being responsive.

[0115] Load balancing decision. The read scheduling module performs load balancing calculations, applicable to scenarios with non-real-time requirements and stateless general query requests. The actions include: collecting real-time metrics from the primary database cluster 31 and the backup database cluster 32: CPU utilization, memory usage, network latency, and current connection count. A weighted round-robin or least-connection algorithm is applied to select the cluster with the lighter current load as the target.

[0116] Routing execution and state maintenance. Requests are forwarded to the identified target database cluster (primary database cluster 31 or backup database cluster 32). For stateful requests, the session-cluster mapping table in gateway device 20 is updated, and an appropriate TTL (Time To Live) is set. Detailed routing logs, including tracing identifiers, are recorded for system monitoring, fault location, and performance analysis.

[0117] Through this three-layer intelligent routing mechanism, the gateway device 20 has achieved a paradigm shift from "cluster level" to "request level", from "coarse-grained" to "fine-grained", and from "passive switching" to "active scheduling". It has truly solved the long-standing technical problem in the industry of being unable to achieve stable load balancing / real-time reading requirements, and provided a solid technical guarantee for high-concurrency and low-latency business scenarios.

[0118] Based on any of the above embodiments, this disclosure also provides a gateway-based distributed data processing device. Figure 8 This is a schematic block diagram illustrating the structure of a gateway-based distributed data processing device according to one embodiment of this disclosure. Figure 8 As shown, the gateway-based distributed data processing device includes: an acquisition module 81, used to acquire data processing requests sent by the service terminal; a determination module 82, used to determine the request type and the corresponding routing rule based on the operation identifier parsed from the data processing request; the determination module 82 is also used to determine the target database cluster from the distributed database based on the request type and the corresponding routing rule; and a forwarding module 83, used to forward the data processing request to the target database cluster for execution; and to feed back the execution result of the target database cluster to the service terminal.

[0119] Module 82 is used to determine the routing rule as a write request if the request type is a write request; and to determine the primary database cluster as the target database cluster from the distributed database based on the write request routing rule.

[0120] The determination module 82 is used to route write requests to the primary database cluster and write the target data; to feed back the execution result of successfully writing the target data to the business side; and to store the target data and timestamp together in a message queue; and to asynchronously replicate the target data to the backup database cluster in the order of timestamps through the message queue.

[0121] The determination module 82 is used to determine the routing rule as a read request routing rule if the request type is a read request; and to determine the primary database cluster or the backup database cluster as the target database cluster from the distributed database according to the read request routing rule.

[0122] The determination module 82 is used to determine whether the read request contains a real-time read identifier; wherein, the real-time read identifier includes at least one of a custom real-time read identifier and a real-time read API; if there is a real-time read identifier, the main database cluster is determined from the distributed database as the target database cluster.

[0123] The determination module 82 is used to parse the read request to obtain the session ID if the read request does not contain a real-time read identifier; and to look up the session cluster mapping table in the gateway device based on the session ID; wherein, the session cluster mapping table is a table in the gateway device used to temporarily cache the mapping relationship between sessions and clusters; if the session ID is found in the session cluster mapping table, the primary database cluster or backup database cluster corresponding to the session ID in the session cluster mapping table is used as the target database cluster.

[0124] The determination module 82 is used to obtain the first load status information of the primary database cluster and the second load status information of the backup database cluster if the session ID is not found in the session cluster mapping table; perform load balancing calculation based on the first load status information and the second load status information; and determine the primary database cluster or the backup database cluster as the target database cluster based on the load balancing calculation result.

[0125] The determination module 82 is used to respond to the upgrade operation of the primary database cluster or the backup database cluster in the distributed database, and to determine whether the version of the data processing request is the same as the database version of the target database cluster. If they are not the same, the protocol format of the data processing request is converted by the protocol adaptation layer in the gateway device to obtain a compatible format data processing request, and the compatible format data processing request is sent to the target database cluster.

[0126] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0127] The executing entity of the gateway-based distributed data processing method in the specific embodiments of this disclosure can be an electronic device such as a server (including a local server or a cloud computing platform).

[0128] Therefore, based on any of the above embodiments, this disclosure also provides an electronic device that can execute the gateway-based distributed data processing method of any of the embodiments described above.

[0129] Figure 9 This is a schematic block diagram of an electronic device according to one embodiment of the present disclosure.

[0130] The hardware architecture of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and overall design constraints. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0131] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0132] This disclosure also provides a readable storage medium storing a computer program that, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection with one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0133] This disclosure also provides a computer program product, the methods of which can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, all or part of the processes or functions of this disclosure are performed.

[0134] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0135] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., refer to specific features, structures, or characteristics described in connection with that embodiment / mode or example, which are included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0140] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0141] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A gateway-based distributed data processing method, characterized in that, The method includes: Get the data processing request sent by the business side; Based on the operation identifier parsed from the data processing request, determine the request type and the routing rule corresponding to the request type; Based on the request type and the corresponding routing rules, the target database cluster is determined from the distributed database; Forward the data processing request to the target database cluster for execution; and The execution results of the target database cluster are fed back to the business side.

2. The gateway-based distributed data processing method according to claim 1, characterized in that, The request type includes: write request; determining the target database cluster from the distributed database according to the request type and routing rules includes: If the request type is a write request, then the routing rule is determined to be a write request routing rule; Based on the write request routing rules, the primary database cluster is determined from the distributed database as the target database cluster.

3. The gateway-based distributed data processing method according to claim 2, characterized in that, The data processing request is forwarded to the target database cluster for execution. And, feeding back the execution results of the target database cluster to the business unit, including: The write request is routed to the main database cluster and the target data is written. The successful execution result of writing the target data is fed back to the business side; and... The target data is combined with a timestamp and then stored in a message queue. The target data is asynchronously replicated to the backup database cluster in timestamp order through the message queue.

4. The gateway-based distributed data processing method according to claim 1, characterized in that, The request type includes: read request; determining the target database cluster from the distributed database according to the request type and routing rules includes: If the request type is a read request, then the routing rule is determined to be a read request routing rule; Based on the read request routing rules, the primary database cluster or backup database cluster is determined from the distributed database as the target database cluster.

5. The gateway-based distributed data processing method according to claim 4, characterized in that, The step of determining the primary database cluster or backup database cluster as the target database cluster from the distributed database according to the read request routing rules includes: Determine whether the read request contains a real-time read identifier; wherein the real-time read identifier includes at least one of a custom real-time read identifier and a real-time read API; If the real-time read identifier is present, then the primary database cluster is determined from the distributed database as the target database cluster.

6. The gateway-based distributed data processing method according to claim 5, characterized in that, The step of determining the primary database cluster or backup database cluster as the target database cluster from the distributed database according to the read request routing rules includes: If the read request does not contain the real-time read identifier, parse the read request to obtain the session ID; The session cluster mapping table is retrieved from the gateway device based on the session ID; wherein, the session cluster mapping table is a table in the gateway device used to temporarily cache the mapping relationship between sessions and clusters; If the session ID is found in the session cluster mapping table, then the primary database cluster or backup database cluster corresponding to the session ID in the session cluster mapping table is taken as the target database cluster.

7. The gateway-based distributed data processing method according to claim 6, characterized in that, The step of determining the primary database cluster or backup database cluster as the target database cluster from the distributed database according to the read request routing rules includes: If the session ID is not found in the session cluster mapping table, then the first load status information of the primary database cluster and the second load status information of the backup database cluster are obtained. Load balancing calculations are performed based on the first load status information and the second load status information, and the primary database cluster or the backup database cluster is determined as the target database cluster based on the load balancing calculation results.

8. The gateway-based distributed data processing method according to claim 1, characterized in that, After determining the target database cluster from the distributed database, the process further includes: In response to an upgrade operation of the primary or backup database cluster in a distributed database, determine whether the version of the data processing request is the same as the database version of the target database cluster. If they are different, the protocol format of the data processing request is converted by the protocol adaptation layer in the gateway device to obtain a compatible format data processing request, which is then sent to the target database cluster.

9. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 8.

Images