Data processing method and system

Abstract

The embodiments of the present specification provide a data processing method and system, wherein the data processing method is applied to a data processing node of a distributed data processing system, and includes: receiving multiple data processing requests sent by a client, determining a target processing quantity based on the quantity of the multiple data processing requests, and performing flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain target data processing requests; forwarding the target data processing requests to a global transaction manager, receiving processing results of the global transaction manager processing the target data processing requests, and returning the processing results to the client corresponding to each data processing request.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and more particularly to a data processing method. One or more embodiments of this specification also relate to a data processing system, a computing device, and a computer-readable storage medium. Background Art

[0002] The database adopts a multi-master MPP (Massive Parallel Processing) architecture. The system consists of two groups of computer nodes: master nodes and data nodes. The database relies on a component called the Global Transaction Manager (GTM) to support snapshot isolation. Although the GTM component can be implemented through multi-process or multi-threading to improve its parallelism, it is essentially centralized, and as the number of concurrent transactions increases, the number of connections to the GTM component will increase significantly, which will not only put a lot of pressure on the operation of the GTM component, but also become a serious system bottleneck for the entire distributed database system. Summary of the invention

[0003] In view of this, the present specification provides a data processing method. One or more embodiments of the present specification also relate to a data processing system, a computing device, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a data processing method is provided, which is applied to a data processing node of a distributed data processing system, including:

[0005] Receive multiple data processing requests sent by a client, determine a target processing quantity based on the quantity of the multiple data processing requests, and perform flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain target data processing requests;

[0006] The target data processing request is forwarded to the global transaction manager, and a processing result of the target data processing request processed by the global transaction manager is received, and the processing result is returned to the client corresponding to each data processing request.

[0007] According to a second aspect of an embodiment of this specification, a data processing system is provided, the data processing system comprising a data processing node, the data processing node comprising:

[0008] a request receiving module, configured to receive a plurality of data processing requests sent by a client, determine a target processing quantity based on the quantity of the plurality of data processing requests, and perform flow limiting processing on the plurality of data processing requests according to the target processing quantity to obtain a target data processing request;

[0009] The proxy module is configured to forward the target data processing request to the global transaction manager, receive the processing result of the global transaction manager on the target data processing request, and return the processing result to the client corresponding to each data processing request.

[0010] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0011] Memory and processor;

[0012] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the data processing method when executing the computer-executable instructions.

[0013] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the data processing method are implemented.

[0014] A data processing method provided by an embodiment of the present specification is applied to a data processing node of a distributed data processing system, including: receiving multiple data processing requests sent by a client, determining a target processing number based on the number of the multiple data processing requests, and performing flow limiting processing on the multiple data processing requests according to the target processing number to obtain target data processing requests; forwarding the target data processing requests to a global transaction manager, receiving a processing result of the global transaction manager processing the target data processing requests, and returning the processing result to the client corresponding to each data processing request.

[0015] Specifically, by performing flow limiting processing on multiple data processing requests sent by the client in the data processing node, all data processing requests sent by the client are processed in batches according to a certain number of data processing requests, which can reduce the request processing pressure of the data processing node. In addition, the subsequent forwarding of the target data processing request to the global transaction manager can also improve the processing efficiency of the global transaction manager and enhance the processing performance of the entire distributed data processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a multi-master distributed database architecture of a data processing method provided by an embodiment of this specification;

[0017] Figure 2 It is an overall architecture diagram of a data processing system provided by an embodiment of this specification;

[0018] Figure 3This is a flow chart of a data processing method provided by one embodiment of this specification applied to a data processing node of a distributed data processing system;

[0019] Figure 4 It is a structural schematic diagram of a shared memory area of ​​a data processing method provided by an embodiment of this specification;

[0020] Figure 5 It is a schematic diagram of various state transitions of a processing slot in a shared memory area of ​​a data processing method provided by an embodiment of the present specification;

[0021] Figure 6 is a structural schematic diagram of a data processing device provided by an embodiment of this specification;

[0022] Figure 7 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0023] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0024] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] First, the terms involved in one or more embodiments of this specification are explained.

[0027] Transaction: In database management systems, a transaction is a single logical unit or work, sometimes consisting of multiple operations. Any logical computation done in a consistent pattern in a database is called a transaction. An example is a transfer of money from one bank account to another: the complete transaction requires subtracting the amount to be transferred from one account and then adding the same amount to another account.

[0028] Database transactions: By definition, a database transaction must be atomic (it must complete completely or have no effects), consistent (it must comply with existing constraints in the database), isolated (it cannot affect other transactions), and durable (it must be written to persistent storage). Database practitioners often use the acronym ACID to refer to these properties of database transactions.

[0029] Distributed database: is a database that stores data in different physical locations. It may be stored in multiple computers located in the same physical location (such as a data center); or it may be scattered across an interconnected computer network.

[0030] Multi-master architecture: is a popular way to build distributed database systems. In this architecture, the distributed database system consists of two groups of computer nodes: master nodes and data nodes. Each master node retains the latest copy of the system catalog and metadata (for example, table and index definitions). The database data is stored in multiple data nodes, subject to user-specified partitioning and / or replication strategies.

[0031] Database session: represents the connection between an application (or client) and the database that stores its persistent objects. The connection is usually established through the TCP network protocol. Once a session is established, the client application can send SQL statements through the connection to the database to query and manipulate the data in the database. In terms of the database system, the database can use a process or thread to receive and service all SQL statements from the session. Once the client application disconnects from the session, the database system de-allocates any resources associated with the session (such as a processing process or thread).

[0032] Multi-version concurrency control (MVCC): Multi-version concurrency control (MCC or MVCC) is a concurrency control method commonly used in database management systems to provide concurrent access to the database. When an MVCC database needs to update a piece of data, it does not overwrite the original data item with the new data, but creates an updated version of the data item. Therefore, multiple versions are stored. The version seen by each transaction depends on the isolation level implemented. The most common isolation level implemented by MVCC is snapshot isolation. With snapshot isolation, the transaction observes the state of the data at the beginning of the transaction.

[0033] Snapshot isolation: Snapshot isolation guarantees that all reads performed within a transaction will see a consistent snapshot of the database (actually it reads the last committed value that existed when it started), and that updates by the transaction can only be successfully committed if there is no conflict between the updates made by the transaction and the updates made by other transactions since the start of the snapshot.

[0034] Proxy (proxy server): In computer networking, a proxy server is a server application or device that acts as an intermediary for client requests seeking resources from servers that provide those resources. The proxy server thus operates on behalf of the client when requesting services, potentially obscuring the true origin of the request to the resource server. Instead of connecting directly to a server that can fulfill the requested resource (such as a file or web page), the client directs the request to a proxy server, which evaluates the request and performs the required network transaction. This is a way to simplify or control the complexity of a request, or to provide additional benefits such as load balancing, privacy, or security.

[0035] The database uses a multi-master MPP (Massive Parallel Processing) architecture. The system consists of two groups of computer nodes: master nodes and data nodes. Each master node retains the latest copy of the system catalog and metadata (for example, table and index definitions). The database data is stored in multiple data nodes, subject to user-specified partitioning and / or replication policies.

[0036] The client connects to one of the hosts (for example, via the TCP network protocol) to establish a database session, and can then submit SQL statements through the connection. For each SQL statement from the client, the corresponding master node parses the SQL statement, generates an optimized query plan, and dispatches the query plan to the data node for execution. Each data node executes the query plan sent by the master node with locally stored data, exchanges intermediate data with each other when necessary, and finally sends the query results back to the master node. The master node merges and assembles the final query results and sends them back to the client.

[0037] For details, please refer to Figure 1 , Figure 1 A multi-master distributed database architecture of a data processing method provided in an embodiment of this specification is shown.

[0038] Figure 1The distributed database architecture is a distributed database with two master database nodes, namely master node 1 and master node 2. Each master node corresponds to two clients. The clients corresponding to master node 1 are client 1 and client 2, and the clients corresponding to master node 2 are client 3 and client 4. At the same time, the data nodes corresponding to the two master nodes are data node 1, data node 2, data node 3, and data node 4, and the multi-master distributed database architecture also has a global transaction manager (GTM).

[0039] When it comes to specific implementation, Figure 1 Each set of rectangles in the table represents a process or thread used to run transactions in a database session. The database supports database transactions that comply with the ACID properties of the SQL standard. In fact, any SQL statement submitted to the database is executed in a database transaction. Such transactions are either explicitly specified by the client through BEGIN / COMMIT / ABORT statements, or implicitly and internally created by the database system for a single SQL statement when the client does not explicitly specify the transaction scope.

[0040] Transactions in a database typically involve a phase of creating and scheduling a query plan for the transaction's SQL statements, and multiple data nodes that execute the queries. If the transaction contains DDL (data definition language) statements (e.g., CREATE TABLE) that modify system catalogs and metadata, the transaction will also span all other master data in the system. Because transactions involve multiple distributed computer nodes, a distributed transaction protocol is used to ensure that the transaction meets ACID properties. In particular, the database adopts a standard two-phase commit protocol. The master node acts as the coordinator of the distributed transaction, and the data nodes (subnodes) involved and other master nodes act as participants. When the work in the transaction is completed, the coordinator initiates the first phase called "prepare" (or "vote"): it asks each participant to vote whether the transaction should be committed; each participant replies with its own vote (commit or abort) based on its local execution results. If the coordinator receives commit votes from all participants, then it initiates the second phase called "commit": it asks all participants to commit the transaction locally, and once all participants confirm that they are done, the coordinator commits the distributed transaction. If the coordinator receives any abort votes from participants during the prepare phase, it asks all participants to abort the transaction locally and aborts the distributed transaction.

[0041] Furthermore, the database relies on a component called the Global Transaction Manager (GTM) to support snapshot isolation, a popular variant of the multi-version concurrency control mechanism for managing concurrent transactions. The GTM is a centralized service in the distributed database that is responsible for assigning unique identifiers to transactions, tracking the state of distributed transactions (whether they are in progress, committed, or aborted), and generating distributed snapshots.

[0042] When a master node starts a distributed transaction, it sends a request to the GTM to register the new transaction with the GTM and assign a unique identifier (called a global transaction ID or GXID for short) to this transaction. The global transaction ID uniquely identifies a transaction. Whenever a transaction inserts or modifies a row in a database table, a version of the row data is stored in the table, which contains the data payload and the transaction ID. The transaction ID is implemented internally as a hidden column that is transparent to the database user.

[0043] After the transaction is registered with GTM, GTM sets its status to in progress. Later when the corresponding master node commits or aborts the transaction, the master node notifies GTM of the change and GTM sets the transaction status accordingly.

[0044] When the master node dispatches a query in a transaction to a data node (Remove), it sends a distributed snapshot request to the GTM. The contents of the distributed snapshot indicate which transactions are in progress at the time (based on their global transaction IDs). To compute the distributed snapshot, the GTM checks its tracking record of transaction status and returns the IDs of all currently active transactions. This distributed snapshot is sent to the data node along with the query plan. When a data node executes a query and needs to access a row in a table, it uses the distributed snapshot to determine whether a version of the row is visible to the current transaction. For example, if the executing transaction ID is listed in the set of in-progress transactions in the distributed snapshot, the current transaction should not read this version, otherwise it will read data from an uncommitted transaction and violate the isolation property.

[0045] Although the GTM component can be implemented as multiple processes or threads to improve its parallelism, it is inherently centralized and will become a serious system bottleneck as the number of concurrent transactions increases. There are two main obstacles to scaling GTM for high concurrency:

[0046] First, in a multi-master distributed database, assuming that each master node can accept up to N client sessions, and there are M master nodes, then the system can have up to N x M concurrent sessions at any time. If each session establishes its own TCP connection to the GTM, the GTM will need to handle up to N x M connections. In large deployments with high concurrency, this may exceed the TCP connection limit of a single machine. Since the whole benefit of having multiple master nodes is to provide clients with scalable connection points beyond the capacity of a single machine, connecting each session to a centralized GTM would defeat the fundamental purpose of a multi-master architecture.

[0047] Second, even if GTM could serve so many connections from a single session, it would lose a lot of opportunities to improve efficiency. For example, suppose there are K concurrent sessions on the same master node, and they are all performing read-only transactions. A read-only transaction is a transaction that does not modify or insert any data. If each session establishes its own connection to GTM and sends a distributed snapshot request from its own connection, GTM will receive K such requests, calculate K distributed snapshots, and send K results back. However, this is redundant and unnecessary, and GTM can use the first distributed snapshot calculated as a response to all K concurrent snapshot requests from these read-only transactions. If the master node can merge the K concurrent requests into a single request, let GTM calculate the distributed snapshot only once, and then take the returned results, we can reduce the number of network messages and the amount of work done by GTM by K times and return them to K transactions. Even for different concurrent requests from the same master node (for example, one session is requesting a global transaction ID and another session is requesting a distributed snapshot), it is usually more efficient to send these requests in batches than to send them individually from separate connections.

[0048] However, for the first problem mentioned above, it may be believed in the prior art that connectionless network protocols such as UDP can overcome the connection limitation, but the result is to increase the complexity of system implementation to ensure reliable communication with GTM. In addition, using UDP does not help solve the second problem at all.

[0049] In the subsequent development of distributed data processing systems, Postgres-XL is a multi-master distributed database that also uses a centralized global transaction manager (GTM). The GTM in Postgres-XL is implemented as a multi-threaded independent process, responsible for transaction ID allocation, transaction status tracking, and distributed snapshot calculation. The user's database session on the master node can directly connect to the GTM to request a transaction ID, notify state changes, or request a distributed snapshot. Alternatively, the database can deploy multiple GTM agent processes and have the user's database session connect to one of the GTM agents. The GTM agent forwards the request to the GTM server and sends the response back to the user session.

[0050] From this, we can see that Postgres-XL uses proxy modules to improve the scalability of centralized GTM, but it also has four obvious defects: First, the GTM proxy in Postgres-XL only supports TCP connections and communications with user database sessions on the master node. Even if the GTM proxy and the user's database session run on the same master node, they communicate with each other through TCP. Due to the additional memory copy and unnecessary network stack overhead, this communication is less efficient than other shared memory-based communication mechanisms. Second, when there are multiple GTM agents, Postgres-XL does not specify how to assign the user's database session to these GTM agents in a load-balanced manner. Third, when the GTM agent in Postgres-XL receives multiple concurrent requests at the same time, it packages these requests into a message and sends it to the GTM server, and decompresses all responses sent back by the GTM server. However, the GTM agent in Postgres-XL does not detect or eliminate redundant concurrent requests (such as multiple distributed snapshot requests from concurrent read-only transactions). Fourth, Postgres-XL does not allow a single database session to choose whether to connect directly or through a GTM proxy to the GTM. This setting is system-wide and cannot be changed dynamically without restarting the database system. This is not flexible and cannot support situations where multiple users need to use different connection methods to GTM at the same time. For example, the database system may restrict ordinary users' database sessions to use GTM proxy, but allow high-priority users or system administrators to use GTM dedicated connections for emergency or maintenance tasks. The data processing method provided in this specification will allow a single user session to dynamically choose whether to connect directly to GTM or use GTM proxy. Compared with Postgres-XL, this method will be more flexible and can better serve mixed use scenarios.

[0051] In summary, the data processing method provided in the embodiments of this specification proposes an agent-based method to solve the scalability challenges of centralized GTM in multi-master distributed databases. It should be noted that the method proposed in this specification is not limited to the specific distributed transaction protocols (such as Two-Phase Commit) and concurrency control mechanisms (such as Snapshot Isolation) used in the database, and can be widely applied to any multi-master distributed database that adopts centralized transaction management.

[0052] In this specification, a data processing method is provided. This specification also relates to a data processing system, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0053] Figure 2The figure shows an overall architecture diagram of a data processing system provided according to one embodiment of the present specification.

[0054] It should be noted that the data processing method provided in the embodiment of this specification proposes to use a proxy module to improve the durability and scalability of the centralized global transaction manager (GTM) in a multi-master distributed database. Specifically, the data processing system includes a data processing node, wherein the data processing node includes: a request receiving module, configured to receive multiple data processing requests sent by a client, determine a target processing quantity based on the number of the multiple data processing requests, and perform flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain a target data processing request; a proxy module, configured to forward the target data processing request to the global transaction manager, and receive the processing result of the global transaction manager processing the target data processing request, and return the processing result to the client corresponding to each data processing request.

[0055] Figure 2 The overall architecture of the distributed data processing system is a distributed database with two master database nodes, namely master node 1 (master1) and master node 2 (master2). Each master node corresponds to two clients. The clients corresponding to master node 1 are client 1 and client 2, and the clients corresponding to master node 2 are client 3 and client 4. At the same time, the data nodes corresponding to the two master nodes are data node 1, data node 2, data node 3, and data node 4, respectively. The multi-master distributed database architecture also has a global transaction manager (GTM). It should be noted that proxy modules are respectively configured in master node 1 and master node 2. At the same time, the module in the master node that receives the data processing request sent by the client and the proxy module are communicated through a shared memory area.

[0056] In practical applications, each master node in a distributed database is equipped with a set of proxy module processes or threads. Each proxy module serves one or more user database sessions on the master database node where it is deployed. Each user database session can dynamically choose to connect to the GTM directly or indirectly through a proxy module on the same master database node. When a database session chooses to use a proxy and needs to make a request to the GTM (for example, to allocate a global transaction ID or obtain a distributed snapshot), it will process the request to the corresponding proxy module and wait for it to return a response. After receiving at least one request from a user database session, if the connection has not been established yet, the proxy module will establish a connection with the GTM and send the request issued by the user database session. The GTM processes each request and sends a response back to the proxy module. For each response received from the GTM, the proxy module determines which database session it belongs to and sends the response back to the database session. The proxy module and the user database session communicate through an efficient shared memory mechanism. Furthermore, when there are multiple proxy modules, our method assigns user sessions to the proxy modules of these GTMs in a load-balanced manner. The proxy module automatically detects and eliminates redundant requests to the GTM by concurrent transactions to reduce network traffic to the GTM and the workload performed by the GTM. Alternatively, this method allows a single user session to dynamically choose whether to connect directly to GTM or use the GTM proxy module, which can more flexibly serve this hybrid usage scenario.

[0057] The data processing system provided by the embodiments of the present specification utilizes a proxy module configured in a master database node in a multi-master distributed database managed by a centralized global transaction manager to perform flow limiting processing on received data processing requests based on an efficient shared memory mechanism, and when multiple proxy modules are configured, the data processing requests are distributed to the proxy modules in a load balancing manner, thereby helping the GTM to reduce redundant data requests, and also reducing the network traffic of global transaction management and the workload performed by the GTM; further, a single user session is allowed to dynamically choose whether to connect directly to the GTM or use the GTM proxy module, which can more flexibly serve this mixed usage scenario.

[0058] See also Figure 3 , Figure 3 A flow chart showing a data processing method provided by one embodiment of the present specification applied to a data processing node of a distributed data processing system is shown, which specifically includes the following steps:

[0059] It should be noted that the data processing method provided in this embodiment is applicable to any multi-master distributed database that adopts centralized transaction management, and no excessive limitations are made here.

[0060] Step 302: Receive multiple data processing requests sent by the client, determine a target processing quantity based on the number of the multiple data processing requests, and perform flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain target data processing requests.

[0061] In specific implementation, the data processing node of the distributed data processing system can be understood as the main node of the distributed database, that is, the data writing node, which can read data, write data, etc., and in the prior art, after the main node receives multiple data processing requests sent by multiple clients, the multiple data processing requests will be directly forwarded to the global transaction manager of the centralized transaction management, and then execute subsequent data transactions. However, since multiple data processing requests are directly connected to the global transaction management, it may cause the system to crash due to too many connections to the global transaction manager; therefore, the data processing method provided in the embodiment of this specification performs flow limiting processing on multiple data processing requests in the main node of the distributed database, so as to reduce the number of data processing requests executed by the global transaction manager.

[0062] In actual applications, after the master node of the distributed database receives multiple data processing requests sent by the client, it will determine the target processing number to be processed at the same time based on the number of multiple data processing requests, and use the target processing number to limit the flow of the above multiple data processing requests, and then filter out the target data processing requests; for example, the master node of the distributed database receives 1000 data processing requests, and based on the 1000 data processing requests, it is determined that the master node can only process 100 data processing requests at the same time. Therefore, the 1000 data processing requests are limited in flow, and only 100 data processing requests are obtained from the 1000 data processing requests in the order of the requests as the target data processing requests.

[0063] Before the master node of the distributed database receives multiple data processing requests from the client, the database system allows each individual user session to dynamically specify whether to use the proxy module or a dedicated link to the GTM to execute transactions in the session; specifically, before receiving the multiple data processing requests sent by the client, the system also includes:

[0064] The request receiving module determines whether to start the proxy module in the data processing node based on preset project requirements, and if so, sends a data processing instruction to the client, wherein the data processing instruction is an instruction to forward the request sent by the client to the proxy module for processing.

[0065] It should be noted that in order to improve data processing capabilities and reduce the transaction processing pressure of the global transaction manager, a distributed database system can configure a proxy module in the master node of the distributed database to proxy multiple data processing requests to reduce the network traffic and execution workload of the global transaction manager.

[0066] During specific implementation, the request receiving module in the main node of the distributed database can also determine whether to start the proxy module configured in the main node according to different project requirements. If it is determined that a large number of data processing requests are required to be processed based on the project requirements, the proxy module can be agreed to start, and the main node can send data processing instructions to the client, wherein the data processing instructions are instructions to forward the request sent by the client to the proxy module for processing.

[0067] In practice, when a database session dynamically changes its connection mode, it will follow one of two steps: If it currently has a direct connection to the GTM and now chooses to use a proxy, it closes the connection to the GTM and acquires a shared memory communication slot to interact with the proxy. If it currently uses a shared memory communication slot and now chooses to connect directly to the GTM, it abandons the shared memory communication slot and establishes a new direct connection to the GTM. The database system may set quotas or limits to restrict which users or sessions can use direct connections to the GTM.

[0068] The data processing method provided in the embodiments of this specification can also dynamically determine whether to use the proxy module configured in the master node based on actual project requirements, thereby improving the processing efficiency of the distributed data processing system.

[0069] Further, the data processing node includes a request receiving module, a shared memory area and a proxy module;

[0070] Before determining the target processing quantity based on the quantity of the plurality of data processing requests, the method further includes:

[0071] The request receiving module establishes a communication connection with the proxy module based on the shared memory area.

[0072] It should be noted that the master node of the distributed database includes a request receiving module, a proxy module and a shared memory area. The proxy module is configured in the master node, and the proxy module can act on behalf of the master node to perform flow limiting processing on data processing requests.

[0073] In actual applications, the request receiving module in the master node can establish a communication connection with the proxy module configured in the master node through the shared memory area, so that the subsequent proxy module can obtain the data processing request sent by the request receiving module from the shared memory area.

[0074] The data processing method provided by the embodiments of this specification is that in a multi-master distributed database using a centralized global transaction manager, each master node can be configured with a proxy module to collect, combine and forward GTM requests from user sessions on the master node, and return the returned responses to the client, and the communication between the proxy module and the user session is run through an efficient shared memory mechanism on the master node to improve the efficiency of data processing.

[0075] Furthermore, the target processing number of the data processing requests processed by the proxy module at the same time can be determined based on the number of processing slots to be allocated in the shared memory area; specifically, determining the target processing number based on the number of the multiple data processing requests includes:

[0076] The request receiving module determines the number of processing slots to be allocated in the shared memory area based on the number of the multiple data processing requests, and uses the number of processing slots to be allocated as the target processing number.

[0077] In actual applications, a distributed database system can allocate a shared memory area on each master node, which is dedicated to the data exchange between the user's database session and the agent. If the database system is implemented using multiple threads in a single process, then this shared memory area can be allocated on the heap memory accessible to all threads in the database system, including the user's database session thread and the agent thread. If the database system is implemented as multiple processes, this shared memory area can be allocated on the shared memory area provided by the underlying operating system and made available to multiple processes.

[0078] In specific implementation, the request receiving module in the master node can determine the number of pending slots that can be allocated to multiple data processing requests in the shared memory area based on the number of multiple data processing requests received. In the initialized state, the number of pending processing slots means the number of data processing requests processed by the proxy module at the same time.

[0079] The data processing method provided in the embodiments of this specification determines the target processing quantity by determining the number of processing slots to be allocated in the shared memory area, so that the subsequent proxy module can perform flow limiting processing on multiple data processing requests based on the target processing quantity.

[0080] In order to enable the proxy module to perform current limiting processing on multiple data processing requests, the data processing requests may be placed in the processing slots of the shared memory area; specifically, performing current limiting processing on the multiple data processing requests according to the target processing quantity to obtain the target data processing request includes:

[0081] The request receiving module places the data processing requests that are equal to the target processing quantity in the to-be-allocated processing slots in the shared memory area according to the target processing quantity;

[0082] The proxy module obtains data processing requests that are the same as the target processing quantity from the processing slots to be allocated, and uses the data processing requests as target data processing requests.

[0083] In actual applications, the request receiving module in the distributed database master node places the data processing requests that are the same as the target processing number in the to-be-allocated processing slots in the shared memory area based on the determined target processing number, indicating that the data processing requests that the proxy module can process at the same time are in the to-be-allocated processing slots, thereby realizing the current limiting processing of multiple data processing requests. Then, the proxy module can obtain the data processing requests that are the same as the target processing number from the to-be-processed slots, and use the obtained data processing requests as the target data processing requests.

[0084] The data processing request provided in the embodiment of this specification places the data processing request in the to-be-allocated processing slot in the shared memory area, and the proxy module obtains the data processing request from the to-be-allocated processing slot, thereby realizing flow limiting processing of multiple data processing requests to improve the data processing efficiency of the distributed data processing system.

[0085] Based on the detailed current limiting process of the proxy module for multiple data processing requests, this embodiment provides a structure diagram of a shared memory area for a data processing method to communicate with a proxy module, which can be seen in Figure 4 , Figure 4 A schematic diagram of the structure of a shared memory area of ​​a data processing method provided in an embodiment of the present specification is shown.

[0086] Figure 4 In the shared memory area, one part is a set of communication slots, each slot in the communication slot is assigned to a user database session and is used for the user database session to exchange GTM requests and responses with the agent. The other part is the agent bookkeeping, and an agent module uses each cell of the array to track various states. The number of communication slots is set according to the maximum number of user database sessions on a single master node, so that each user database session can be guaranteed to have a dedicated shared memory communication slot.

[0087] In actual applications, there may be more processing slots in the shared memory area, but the processing slots need to be configured before they can be used, and some resource costs are required during the configuration process. Therefore, how to efficiently use the configured processing slots is an effective way to save resource costs. Furthermore, the method adopted in this embodiment is to determine the state of the processing slot based on the signal amount in the processing slot, whether it is occupied. After the proxy module obtains the data processing request from the processing slot, the processing slot is restored to an idle state, and the next data processing request can be allocated to the processing slot; specifically, the request receiving module places the data processing request that is the same as the target processing number in the to-be-allocated processing slot of the shared memory area according to the target processing number, including:

[0088] The request receiving module determines the semaphore of each to-be-allocated processing slot of the shared memory area based on the target processing quantity;

[0089] Based on the semaphore, data processing requests equal to the target processing quantity are placed in the to-be-allocated processing slots of the shared memory area, and the communication state in the to-be-allocated processing slots is modified.

[0090] It should be noted that the proxy module uses semaphores to notify the user database session about slot status changes. For example, the proxy module will use credits to wake up the user session waiting for a response to be ready. It should be noted that the semaphore may be different during each data exchange. For example, if the proxy module and the user database session are separate processes, it can be implemented using the POSIX semaphore system call, or if the proxy module and the user database session are different threads in the same process, it can be implemented as a pthead condition variable.

[0091] In specific implementation, the request receiving module of the distributed database master node can determine the semaphore of each to-be-allocated processing slot in the shared memory area based on the target processing quantity, and place the data processing requests identical to the target processing quantity in the to-be-allocated processing slot in the shared memory area based on the semaphore. At the same time, the communication status of the to-be-allocated processing slot also needs to be modified to facilitate the subsequent determination of whether the processing slot can place the data processing request based on the communication status of the to-be-allocated processing slot.

[0092] In practice, the slot state represents the current state of the slot. During the communication between the user database session and the proxy module, the slot will switch between various states. Figure 5 , Figure 5 A schematic diagram showing various state transitions of processing slots in a shared memory area of ​​a data processing method provided by an embodiment of this specification is shown.

[0093] Figure 5 The figure shows the transition diagram of various slot states, including FREE (the slot is not used by any user database session), EMPTY (the slot has been allocated to a database user session, but no request has been stored yet), REQUEST_AVAIL (the request has been stored in the slot), WAIT_RESPONSE (the agent has sent a request to the GTM and is waiting for a response from the GTM), RESPONSE_AVAIL (the response to the unfinished request has been stored in the slot and is ready for the database user session to receive it), ERROR (an error occurred while serving the current request), and FREEING (the slot has been abandoned by the database session and is ready to be recycled).

[0094] This embodiment can take the process of a user database session acquiring a shared memory communication slot, the process of a user database session giving up a communication slot, and the process of a user database session sending a request and receiving a response through a shared memory communication slot as examples to illustrate the process of slot state conversion:

[0095] The process of a user database session acquiring a shared memory communication slot is as follows: Step 1: Acquire the exclusive lock of the communication slot array; Step 2: Traverse the communication slot array and find a slot with a status of FREE. If found, execute step 3. If the slot is not found, execute step 7; Step 3: Change the slot's status to EMPTY; Step 4: Assign an agent to serve this slot according to the load balancing strategy described in the next section, and add the array index of this slot to the agent's slot index array. Step 5: Release the exclusive lock on the communication slot array; Step 6: Arrange a callback function that will be executed at the end of this session to abandon the acquired slot; Step 7: Return whether the communication slot was successfully acquired.

[0096] Once a communication slot is acquired, the user database session will continue to use it until the session ends. The process of a user database session giving up a communication slot is as follows: Step 1: Change the state of the slot to FREEING; Step 2: Set a semaphore in the corresponding agent's accounting data to notify the agent of the change in the slot state; The agent will handle the recycling of the slot.

[0097] The process of a user database session sending a request and receiving a response through a shared memory communication slot is as follows: Step 1: Store the request in the Request buffer of the communication slot. The request is formatted and serialized into a continuous sequence of bytes, and the index of the communication slot is stored as part of the request; Step 2: Change the slot state to REQUEST_AVAIL; Step 3: Set a Semaphore in the corresponding proxy's accounting data to notify the proxy of the arrival of a new request; Step 4: Wait for the semaphore in the communication slot to obtain a notification from the proxy; Step 5: After receiving the Semaphore notification from the proxy module, check the slot state: (1) If the state is RESPONSE_AVAIL, it means that there is a response in the Response buffer of the slot; in this case, change the slot state to EMPTY and return the response. (2) If the state is ERROR, change the slot state to EMPTY and return an error. (3) If the state is REQUEST_AVAIL or WAIT_RESPONSE, return to step 4 and continue waiting.

[0098] It should be noted that the data processing request is a large buffer that can store a single request to the GTM. When the user database session needs to make a request to the GTM, it constructs the request and stores it in the request buffer. Then the proxy module reads the request from the buffer and sends it to the GTM for processing. The user database session can format and serialize the request into a specific format. Another buffer for the data response is able to store a single response from the GTM. When the proxy module receives a response from the GTM, it stores the response in the buffer and notifies the user database session of its availability through the semaphore of the slot. The shared memory communication slot array is protected by a read-write lock, which itself also resides in the shared memory area. When the user database session needs to acquire a slot, it acquires this lock in write (or exclusive) mode and selects a free slot for use. Each proxy repeatedly scans the communication slots to find unfinished requests and time slots that have been abandoned by the database session and should therefore be reclaimed; the proxy acquires this lock in read (or shared) mode before scanning.

[0099] The data processing method provided in the embodiments of this specification can determine the current state of the slot through the semaphore, and subsequently record the communication state of the slot to achieve efficient allocation of data processing requests, thereby facilitating subsequent improvement in the efficiency of processing data processing requests.

[0100] In addition, the shared memory area also has a portion of bookkeeping data, which records the state of the corresponding proxy module, such as whether it is connected or disconnected with the proxy module, and also records the index data of the slot, recording how many slots are used and how many slots are idle, that is, the bookkeeping data in the shared memory area records the state of the proxy module itself; specifically, after the request receiving module places the data processing request identical to the target processing number in the to-be-allocated processing slot of the shared memory area according to the target processing number, it also includes:

[0101] The request receiving module records the status of the to-be-allocated processing slot in the proxy book in the shared memory area based on the data processing request having the same number as the target processing quantity, and records the connection status with the proxy module.

[0102] In practical applications, the proxy bookkeeping can be understood as bookkeeping data (per-proxyData), and the number of units in the bookkeeping data is set according to the number of proxy modules on the master node. The bookkeeping data of each proxy module includes the following fields: the user database session uses a semaphore to notify the proxy module of the arrival of a new data processing request or the abandonment of a communication slot. Upon receiving the notification, the proxy module starts scanning the communication sequence to work; the proxy identifier (proxy_id) is the unique identifier of the proxy that owns this bookkeeping data; the slot index data is the index array of the communication slots served by the proxy module. When there are multiple proxy modules on the master node, each proxy module will serve a subset of slots, and its slot index array identifies these slots.

[0103] In specific implementation, the request receiving module records the status of the processing slot to be allocated in the proxy book of the shared memory area based on the data processing request with the same target processing quantity, and at the same time, the status of the proxy module can also be recorded.

[0104] The data processing method provided in the embodiment of the present specification records the status of the slot and the status of the proxy module in the proxy book of the shared memory area. Based on the conversion and recording of the status, it can not only reduce the conflict of concurrent access, but also reduce resource overhead.

[0105] Step 304: forward the target data processing request to the global transaction manager, receive the processing result of the global transaction manager on the target data processing request, and return the processing result to the client corresponding to each data processing request.

[0106] In actual applications, the distributed data processing node forwards the determined target data processing request to the global transaction manager. After the global transaction manager processes the data processing request, it returns the processing result of the target data processing request to the distributed data processing node, and returns the processing result to the client corresponding to each data processing request.

[0107] Further, forwarding the target data processing request to the global transaction manager includes:

[0108] When determining to establish a communication connection with the global transaction manager, the proxy module forwards the target data processing request to the global transaction manager based on the communication connection.

[0109] In actual applications, when the proxy module in the master node of the distributed database determines that a communication connection has been established with the global transaction manager, it forwards the target data processing request to the global transaction manager through the communication connection.

[0110] After determining that the type of the target data processing request is a distributed snapshot request, the request receiving module may also receive a result of calculating the distributed snapshot request returned by the global transaction manager; specifically, forwarding the target data processing request to the global transaction manager and receiving a processing result of the target data processing request processed by the global transaction manager includes:

[0111] The request receiving module forwards the distributed snapshot request to the global transaction manager when determining that the target data processing request is a distributed snapshot request;

[0112] The request receiving module receives a result of the global transaction manager calculating each distributed snapshot request.

[0113] In actual applications, after the request receiving module determines that the target processing request is a distributed snapshot request, the distributed snapshot request is sent to the GTM, and the GTM processes the request by calculating the distributed snapshot and replies to the proxy module with a response containing the distributed snapshot and the slot index list contained in the combined request. After receiving the response, the proxy module stores a copy of the distributed snapshot in the response buffer of each communication slot contained in the response.

[0114] Furthermore, this embodiment also supports forwarding distributed snapshot requests to the global transaction manager using a hybrid communication mode. Step 1: Given a group of new requests in the communication slots in the waiting queue, the proxy will check them and find all requests for obtaining distributed snapshots initiated for read-only transactions, as well as requests for obtaining transaction IDs initiated for any type of transaction. Step 2: If the group of transactions currently being processed does contain multiple requests for obtaining distributed snapshots initiated for read-only transactions, the proxy will merge them and construct a new joint request. Its type is still a distributed snapshot request, and it contains a slot index list indicating which communication slot each original request originated from. The proxy sends this new joint request to the GTM. After the GTM calculates the distributed snapshot, it returns it and the received slot index list as a response to the proxy. After receiving the response from the GTM, the proxy copies the distributed snapshot in the message to the result data buffer of each communication slot pointed to by the slot index. Step 3: Similarly, if the group of transactions currently being processed does contain multiple requests for obtaining transaction IDs, the proxy will also merge them and construct a new joint request, whose type is still a global transaction ID request, and contains a slot index list to indicate which communication slot they all come from. After this joint request is sent to the GTM, the GTM allocates a group of continuous transaction IDs and returns the range information of this continuous transaction ID and the slot index list to the proxy. After receiving the response, the proxy writes each transaction ID in the range one by one into the data buffer of each index pointed to by the slot index list.

[0115] In addition, a plurality of proxy modules may be configured in the master node of the distributed database; specifically, the request receiving module establishes a communication connection with the proxy module based on the shared memory area, including:

[0116] The request receiving module determines two or more proxy modules that are equal to the preset configuration quantity based on the preset configuration quantity, and establishes communication connections with the two or more proxy modules based on the shared memory area.

[0117] In practice, the number of proxy modules on the master node is configurable, and when there are multiple proxies, our approach distributes user sessions to these GTM proxies in a load-balanced manner. Assume that there are K proxies on each master server. Whenever a user database session needs to obtain a communication slot, it scans the communication slot array from the beginning (i.e., the first slot in the array) to find a slot with an idle status. It then determines which proxy serves this slot by calculating: proxy id = slotindex modulo K (i.e., the number of proxies); in other words, slots are allocated to proxies in a static round-robin manner. This allocation is simple, deterministic, and generally results in an even distribution of work to proxies. More complex alternative allocation strategies are possible. For example, a session might check and compare the lengths of the slot index arrays of the proxies and select the one that serves the least number of slots.

[0118] Furthermore, when there are multiple proxy modules, this embodiment proposes a load balancing method to distribute user sessions to multiple proxy modules; specifically, forwarding the target data processing request to the global transaction manager includes:

[0119] The request receiving module distributes the target data processing request to the two or more proxy modules based on a preset load balancing mechanism, and the proxy modules forward the target data processing request to the global transaction manager.

[0120] In actual applications, after the request receiving module distributes the target data processing request to two or more proxy modules based on the preset load balancing mechanism, the proxy module automatically detects and eliminates redundant requests from concurrent transactions to GTM to reduce network traffic to GTM and the workload performed by GTM. The proxy checks two types of concurrent requests that can be served together and can therefore be considered redundant. The first type is a distributed snapshot request from a concurrent read-only transaction. GTM can calculate a single distributed snapshot and return it as a response to all these requests. Therefore, the proxy first presents multiple concurrent distributed snapshot requests as a single combined message to GTM, and after receiving the response, returns a copy of the response to each request transaction. The second type is to request a global transaction ID from a concurrent transaction. Instead of assigning a separate ID to each transaction and having the proxy process one for each request transaction, GTM can return a continuous range of global transaction IDs.

[0121] The process of message reduction and grouping by the proxy module is as follows: Step 1: Given a set of new requests in the communication slots in the waiting queue, the proxy will check them and find all requests for obtaining distributed snapshots initiated for read-only transactions and requests for obtaining transaction IDs initiated for any type of transaction. Step 2: If the current set of transactions does contain multiple requests for obtaining distributed snapshots initiated for read-only transactions, the proxy will merge them and construct a new joint request. Its type is still a distributed snapshot request and contains a slot index list indicating which communication slot each original request originated from. The proxy sends this new joint request to the GTM. After the GTM calculates the distributed snapshot, it returns it and the received slot index list as a response to the proxy. After receiving the response from the GTM, the proxy copies the distributed snapshot in the message to the result data buffer of each communication slot pointed to by the slot index. Step 3: Similarly, if the current set of transactions does contain multiple requests for obtaining transaction IDs, the proxy will also merge them and construct a new joint request, whose type is still a global transaction ID request and contains a slot index list indicating which communication slot they all originated from. After this joint request is sent to GTM, GTM allocates a set of continuous transaction IDs and returns the range information of this continuous transaction ID and the slot index list to the proxy. After receiving the response, the proxy writes each transaction ID in the range one by one into the data buffer of each index pointed to by the slot index list.

[0122] Furthermore, in the embodiments of this specification, each proxy module provides services for one or more user database sessions through their communication slots, wherein the workflow of the proxy module is as follows:

[0123] Step 1: Wait for the semaphore of the agent's bookkeeping data to get notification from the user database session. After receiving the notification, go to step 2; Step 2: Get a shared lock on the communication slot array; Step 3: Scan each slot whose index is recorded in the agent's slot index array. These are the slots that serve this agent. For each such slot, check its state. If the state is FREEING, add the slot to a list called freeing_slots; If the state is REQUEST_AVAIL, add the slot to another list called pending_requests; Step 4: For each slot in the freeing_slots list, reclaim it by changing its state to FREE, or delete this slot from the agent's slot index array; Step 5: Unlock the communication slot array; Step 6: Send a request from the slot in the pending_requests list: Step 6.1: If there are multiple pending requests, merge them into one or more combined requests and send the combined message to the GTM. Otherwise, send a single request to the GTM. The details of building a combined request will be described later. After sending the request, change the slot state to WAIT_RESPONSE. If the proxy fails to send a request, change the state of the corresponding slot to ERROR, remove these slots from the pending_requests list, reestablish the connection with the GTM, and go to step 1. Step 6.2: Receive a response from the GTM. The GTM may return a combined response corresponding to multiple requests, or a single response. For a combined response, build a single response from it. For each response, store it in the response buffer of the corresponding communication slot and change the slot state to RESPONSE_AVAIL. If the proxy does not receive a response from the GTM (for example, due to a network disconnect or GTM failure), set the corresponding slot state to ERROR. Once the state of the slot is changed to RESPONSE_AVAIL or ERROR, set the slot's semaphore to notify the database session and remove the slot from the pending_requests list. When the pending_requests list is empty, go to step 1.

[0124] Based on this, both the database session and the agent modify the slot state through atomic instructions (e.g., compare and exchange). It is important to note that the recycling of a communication slot is done in two steps: the session sets the slot state to FREEING, and then the agent reclaims it and sets its state to FREE. This arrangement reduces conflicts for concurrent access to the communication slot array, thereby reducing locking overhead.

[0125] In summary, the data processing method provided by the embodiments of this specification is a multi-master distributed database using a centralized global transaction manager, in which each master node has a configurable number of proxy modules for collecting, combining and forwarding GTM requests from the user session, and routing the returned responses back to the user session. The communication between the proxy module and the user session is carried out through an efficient shared memory mechanism on the master server; the number of proxy modules on a master is configurable. When there are multiple proxy modules, our method assigns user sessions to these proxy modules in a load-balanced manner; the proxy automatically detects and eliminates redundant requests to the GTM by concurrent transactions to reduce network traffic to the GTM and the workload performed by the GTM; the database system allows each individual user session to dynamically specify whether to use a proxy or a dedicated connection to the GTM to execute transactions in the session.

[0126] Corresponding to the above method embodiment, this specification also provides a data processing device embodiment, Figure 6 FIG. 1 is a schematic diagram showing the structure of a data processing device provided by an embodiment of the present specification. Figure 6 As shown, the device is applied to a data processing node of a distributed data processing system, comprising:

[0127] The request receiving module 602 is configured to receive multiple data processing requests sent by the client, determine a target processing quantity based on the number of the multiple data processing requests, and perform flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain a target data processing request;

[0128] The proxy module 604 is configured to forward the target data processing request to the global transaction manager, receive the processing result of the global transaction manager on the target data processing request, and return the processing result to the client corresponding to each data processing request.

[0129] The request receiving module 602 is further configured to establish a communication connection with the proxy module based on the shared memory area.

[0130] The request receiving module 602 is further configured to determine the number of processing slots to be allocated in the shared memory area based on the number of the multiple data processing requests, and use the number of processing slots to be allocated as the target processing number.

[0131] The request receiving module 602 is further configured to place data processing requests equal to the target processing quantity in the to-be-allocated processing slots of the shared memory area according to the target processing quantity;

[0132] The proxy module 604 is further configured to obtain data processing requests that are the same as the target processing quantity from the processing slots to be allocated, and use the data processing requests as target data processing requests.

[0133] The proxy module 604 is further configured to forward the target data processing request to the global transaction manager based on the communication connection when it is determined that a communication connection is established with the global transaction manager.

[0134] The request receiving module 602 is further configured to determine the semaphore of each to-be-allocated processing slot of the shared memory area based on the target processing quantity;

[0135] Based on the semaphore, data processing requests equal to the target processing quantity are placed in the to-be-allocated processing slots of the shared memory area, and the communication state in the to-be-allocated processing slots is modified.

[0136] The request receiving module 602 is further configured to record the status of the to-be-allocated processing slot in the proxy book in the shared memory area and the connection status with the proxy module based on the data processing request that is the same as the target processing quantity.

[0137] The request receiving module 602 is further configured to forward the distributed snapshot request to the global transaction manager when determining that the target data processing request is a distributed snapshot request;

[0138] The request receiving module 602 is further configured to receive a result of the global transaction manager calculating each distributed snapshot request.

[0139] The request receiving module 602 is further configured to determine two or more proxy modules that are equal to the preset configuration quantity based on the preset configuration quantity, and establish communication connections with the two or more proxy modules based on the shared memory area.

[0140] The request receiving module 602 is further configured to distribute the target data processing request to the two or more proxy modules based on a preset load balancing mechanism, and the proxy modules forward the target data processing request to the global transaction manager.

[0141] The request receiving module 602 is further configured to determine whether to start the proxy module in the data processing node based on preset project requirements, and if so, send a data processing instruction to the client, wherein the data processing instruction is an instruction to forward the request sent by the client to the proxy module for processing.

[0142] The data processing device provided in the embodiments of the present specification can reduce the request processing pressure of the data processing node by performing flow limiting processing on multiple data processing requests received from the client in the data processing node, and processing all data processing requests sent by the client in batches according to a certain number of data processing requests. In addition, the subsequent forwarding of the target data processing request to the global transaction manager can also improve the processing efficiency of the global transaction manager and enhance the processing performance of the entire distributed data processing system.

[0143] The above is a schematic scheme of a data processing device of this embodiment. It should be noted that the technical scheme of the data processing device and the technical scheme of the above data processing method belong to the same concept, and the details of the technical scheme of the data processing device that are not described in detail can be referred to the description of the technical scheme of the above data processing method.

[0144] Figure 7 The block diagram of a computing device 700 according to an embodiment of the present specification is shown. The components of the computing device 700 include but are not limited to a memory 710 and a processor 720. The processor 720 is connected to the memory 710 via a bus 730, and the database 750 is used to store data.

[0145] The computing device 700 also includes an access device 740 that enables the computing device 700 to communicate via one or more networks 760. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 740 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0146] In one embodiment of the present specification, the above components of the computing device 700 and Figure 7 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 7 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0147] The computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 700 may also be a mobile or stationary server.

[0148] The processor 720 is used to execute the following computer executable instructions, wherein the processor is used to execute the computer executable instructions, wherein the processor implements the steps of the data processing method when executing the computer executable instructions.

[0149] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above data processing method.

[0150] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the data processing method when executed by a processor.

[0151] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above data processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above data processing method.

[0152] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0153] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0154] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0156] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A data processing method, applied to a data processing node of a distributed data processing system, wherein the data processing node comprises a request receiving module, a shared memory area and a proxy module, including: Receive multiple data processing requests sent by the client, determine a target processing quantity based on the quantity of the multiple data processing requests, and perform flow limiting processing on the multiple data processing requests according to the target processing quantity to obtain target data processing requests, wherein the target data processing requests are data processing requests obtained by the proxy module from the processing slot to be allocated and are the same as the target processing quantity, and the data processing requests in the processing slot to be allocated are placed by the request receiving module in the processing slot to be allocated in the shared memory area; The target data processing request is forwarded to the global transaction manager, and a processing result of the target data processing request processed by the global transaction manager is received, and the processing result is returned to the client corresponding to each data processing request, wherein the target data processing request is a distributed snapshot request, and the processing result is a result of calculating the distributed snapshot request returned by the global transaction manager and received by the request receiving module.

2. The data processing method according to claim 1, before determining the target processing quantity based on the quantity of the plurality of data processing requests, further comprising: The request receiving module establishes a communication connection with the proxy module based on the shared memory area.

3. The data processing method according to claim 2, wherein determining the target processing quantity based on the quantity of the plurality of data processing requests comprises: The request receiving module determines the number of processing slots to be allocated in the shared memory area based on the number of the multiple data processing requests, and uses the number of processing slots to be allocated as the target processing number.

4. The data processing method according to claim 3, wherein the step of performing flow limiting processing on the plurality of data processing requests according to the target processing quantity to obtain the target data processing request comprises: The request receiving module places the data processing requests that are equal to the target processing quantity in the to-be-allocated processing slots in the shared memory area according to the target processing quantity; The proxy module obtains data processing requests that are the same as the target processing quantity from the processing slots to be allocated, and uses the data processing requests as target data processing requests.

5. The data processing method according to claim 2 or 4, wherein forwarding the target data processing request to a global transaction manager comprises: When determining to establish a communication connection with the global transaction manager, the proxy module forwards the target data processing request to the global transaction manager based on the communication connection.

6. The data processing method according to claim 4, wherein the request receiving module places the data processing requests that are equal to the target processing quantity in the to-be-allocated processing slots in the shared memory area according to the target processing quantity, comprising: The request receiving module determines the semaphore of each to-be-allocated processing slot of the shared memory area based on the target processing quantity; Based on the semaphore, data processing requests equal to the target processing quantity are placed in the to-be-allocated processing slots of the shared memory area, and the communication state in the to-be-allocated processing slots is modified.

7. The data processing method according to claim 6, after the request receiving module places the data processing requests having the same number of data processing as the target processing number in the to-be-allocated processing slots in the shared memory area according to the target processing number, further comprising: The request receiving module records the status of the to-be-allocated processing slot in the proxy book in the shared memory area based on the data processing request that is the same as the target processing quantity, and records the connection status with the proxy module.

8. The data processing method according to claim 7, wherein forwarding the target data processing request to a global transaction manager and receiving a processing result of the target data processing request processed by the global transaction manager comprises: The request receiving module forwards the distributed snapshot request to the global transaction manager when determining that the target data processing request is a distributed snapshot request; The request receiving module receives the result of the global transaction manager calculating each distributed snapshot request.

9. The data processing method according to claim 2, wherein the request receiving module establishes a communication connection with the proxy module based on a shared memory area, comprising: The request receiving module determines two or more proxy modules that are equal to the preset configuration quantity based on the preset configuration quantity, and establishes communication connections with the two or more proxy modules based on the shared memory area.

10. The data processing method according to claim 9, wherein forwarding the target data processing request to a global transaction manager comprises: The request receiving module distributes the target data processing request to the two or more proxy modules based on a preset load balancing mechanism, and the proxy modules forward the target data processing request to the global transaction manager.

11. The data processing method according to claim 10, before receiving a plurality of data processing requests sent by the client, further comprising: The request receiving module determines whether to start the proxy module in the data processing node based on preset project requirements, and if so, sends a data processing instruction to the client, wherein the data processing instruction is an instruction to forward the request sent by the client to the proxy module for processing.

12. A data processing system, comprising a data processing node, wherein the data processing node comprises a request receiving module, a shared memory area and a proxy module: a request receiving module, configured to receive a plurality of data processing requests sent by a client, determine a target processing quantity based on the quantity of the plurality of data processing requests, and perform flow limiting processing on the plurality of data processing requests according to the target processing quantity to obtain a target data processing request; The proxy module is configured to forward the target data processing request to the global transaction manager, receive the processing result of the global transaction manager on the target data processing request, and return the processing result to the client corresponding to each data processing request, in, The target data processing request is a data processing request obtained by the proxy module from the processing slot to be allocated and having the same number as the target processing request, and the data processing request in the processing slot to be allocated is placed by the request receiving module in the processing slot to be allocated in the shared memory area. The target data processing request is a distributed snapshot request. When it is determined that the target data processing request is the distributed snapshot request, the processing result is the result of calculating the distributed snapshot request returned by the global transaction manager and received by the request receiving module.

13. A computing device comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, wherein the processor implements the steps of the data processing method according to any one of claims 1 to 11 when executing the computer-executable instructions.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images