Data processing method and data processing apparatus

Abstract

The application relates to the technical field of databases, in particular to a data processing method and a data processing device, the data processing method comprising the following steps: when it is determined that a data processing request is a data reading request, first polling a plurality of slave servers and determining one of the slave servers as a target slave server, obtaining a first log sequence number stored in the target slave server, then obtaining a second log sequence number stored in a master server, and determining a synchronization delay difference between the master server and the target slave server according to the first log sequence number and the second log sequence number; when the synchronization delay difference is less than or equal to a preset threshold value and the polling frequency is less than or equal to a preset maximum polling frequency, stopping the polling and distributing the data reading request to the target slave server, and returning a query result obtained by the target slave server according to the data reading request; the query result obtained in this way guarantees that consistency and real-time performance both meet preset requirements, and the accuracy of the query result is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data processing method and a data processing apparatus. Background Technology

[0002] To improve the high availability, scalability, and concurrency capabilities of databases, database clustering technology was developed. A database cluster typically includes multiple server nodes, each consisting of a master server and multiple slave servers. The master and slave servers store the same data and communicate and synchronize information via a network to maintain data consistency and availability.

[0003] To accommodate the read-heavy, write-light business requirements in most application scenarios, the master server typically handles write operations, while the slave server handles read operations. However, due to synchronization delays or other factors between the master and slave servers, the data stored on the slave server may not be up-to-date. This can lead to inconsistencies between query results retrieved from a slave server and those stored on the master server, thus affecting the accuracy of the query results. Summary of the Invention

[0004] This invention provides a data processing method and a data processing apparatus to solve the technical defect in the prior art where the query results obtained from the server are inconsistent with the query results stored on the main server.

[0005] On one hand, the present invention provides a data processing method applied in a distributed database cluster, the distributed database cluster including multiple server nodes, each server node including a master server and multiple slave servers; for any one of the server nodes, the data processing method includes:

[0006] In response to a received data processing request, when the data processing request is determined to be a data read request, the following steps are executed in a loop;

[0007] The polling step involves polling the multiple slave servers and determining one of them as the target slave server, and obtaining the first log sequence number stored in the target slave server determined this time.

[0008] The synchronization delay difference determination step involves obtaining the second log sequence number stored in the master server and determining the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number.

[0009] The polling stop determination step determines whether the synchronization delay difference is less than or equal to a preset threshold and whether the number of polls is less than or equal to a preset maximum number of polls. If so, the polling is stopped and the data reading request is allocated to the target slave server, and the query result obtained by the target server based on the data reading request is returned.

[0010] If not, increase the number of polling iterations and repeat the above polling steps, synchronization delay difference determination steps, and polling stop determination steps until polling stops.

[0011] Optionally, the polling stop determination step further includes:

[0012] If the synchronization delay difference is greater than a preset threshold and the number of polling is equal to the preset maximum number of polling, then polling is stopped and the data read request is assigned to the master server; the query result obtained by the master server based on the data read request is returned.

[0013] Optionally, the data processing method further includes:

[0014] In response to a received data processing request, if the data processing request is determined to be a data write request, the data write request is assigned to the master server, and the second log sequence number stored in the master server is returned.

[0015] Optionally, determining the synchronization latency difference between the master server and the target slave server based on the first log sequence number and the second log sequence number includes:

[0016] The synchronization latency difference between the master server and the slave server is determined by the following formula:

[0017]

[0018] in, The synchronization latency difference between the master server and the slave server, M LNN The second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

[0019] On the other hand, this application also provides a data processing device applied in a distributed database cluster, the distributed database cluster including multiple server nodes, each server node including a master server and multiple slave servers; the data processing device includes: a request receiving unit, a polling unit, a synchronization delay difference determining unit, a polling stop determining unit, and a polling control unit;

[0020] The request receiving unit is used to respond to the received data processing request and, when determining that the data processing request is a data reading request, trigger the polling unit, the synchronization delay difference determination unit, the polling stop determination unit, and the polling control unit to work.

[0021] The polling unit is used to poll the plurality of slave servers and determine one of the slave servers as the target slave server, and obtain the first log sequence number stored in the target slave server determined this time.

[0022] A synchronization delay difference determination unit is used to obtain the second log sequence number stored in the master server and determine the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number.

[0023] The polling stop determination unit is used to determine whether the synchronization delay difference is less than or equal to a preset threshold and whether the number of polling is less than or equal to a preset maximum number of polling. If so, the polling is stopped and the data reading request is allocated to the target slave server, and the query result obtained by the target server according to the data reading request is returned.

[0024] The polling control unit is used to control the polling unit, the synchronization delay difference determination unit, and the polling stop determination unit to work in a loop until polling stops when the number of polling attempts is less than the preset maximum number of polling attempts and the synchronization delay difference is greater than a preset threshold.

[0025] Optionally, the polling stop determination unit is further configured to:

[0026] If the synchronization delay difference is greater than a preset threshold and the number of polling is equal to the preset maximum number of polling, then polling is stopped and the data read request is assigned to the master server; the query result obtained by the master server based on the data read request is returned.

[0027] Optionally, the request receiving unit is further configured to, in response to a received data processing request, allocate the data writing request to the master server when it is determined that the data processing request is a data writing request, and return the second log sequence number stored in the master server.

[0028] Optionally, the synchronization delay difference determination unit is specifically used for:

[0029] The synchronization latency difference between the master server and the slave server is determined by the following formula:

[0030]

[0031] in, The synchronization latency difference between the master server and the slave server, M LSNThe second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the data processing methods described above.

[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above.

[0034] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data processing methods described above.

[0035] The data processing method provided by this invention, when determining that the data processing request is a data read request, first polls multiple slave servers and determines one of them as the target slave server, obtains the first log sequence stored in the target slave server, and then obtains the second log sequence number stored in the master server. Based on the first log sequence number and the second log sequence number, the synchronization latency difference between the master server and the target slave server is determined. When it is determined that the synchronization latency difference is less than or equal to a preset threshold and the number of polls is less than or equal to a preset maximum number of polls, it indicates that the latency difference between the data information stored in the target slave server determined in this poll and the data stored in the master server is small or non-existent. Polling is stopped, and the data read request is allocated to the target slave server. The query result obtained by the target slave server according to the data read request is returned. This query result is consistent with the query result stored in the master server, ensuring the accuracy of the query result. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0037] Figure 1 One of the schematic flowcharts of a data processing method is provided for an embodiment of the present invention;

[0038] Figure 2 A second schematic diagram of a data processing method is provided for an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of a data processing device structure provided in an embodiment of the present invention;

[0040] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0042] To help those skilled in the art better understand the technical solution of this application, the following explains some of the technical terms involved in this application.

[0043] Database clustering: Database clustering refers to combining multiple database servers to collectively handle database workloads. It improves database performance, scalability, and reliability by distributing data and workloads across multiple servers. Furthermore, database clusters can expand system processing power and storage capacity by adding more nodes. New nodes can balance the load and increase system throughput to meet ever-growing data demands. In addition, database clusters provide highly available database services through redundancy backup and failover mechanisms. When one node fails, other nodes can take over its work, ensuring database continuity. Common database cluster implementations include master-slave replication, multi-master replication, and shared storage. This invention primarily focuses on database clusters based on master-slave replication, the most common cluster configuration, where one server acts as the master server, handling write operations, while the other servers act as slave servers, handling read operations.

[0044] Database cluster proxy: This refers to a middleware component located in front of a database cluster, used to manage and route database requests. It acts as a proxy between the application and the database cluster, handling functions such as request forwarding, load balancing, failover, and query optimization. Common proxies include Pgpool and HAProxy.

[0045] Read operations refer to the operations of retrieving data from a database. They include query statements (such as SELECT) and the process of reading data. Read operations are typically used to obtain data for application use and do not modify the data in the database.

[0046] Write operations refer to operations that modify data in a database. These include operations such as INSERT, UPDATE, and DELETE. Write operations are used to add new data, update existing data, or delete data in the database.

[0047] Synchronization delay: In database master-slave replication, due to factors such as network latency, server load, and replication mechanism, the data on the slave server may not be able to keep synchronized with the data on the master server immediately, resulting in a time difference between the data on the slave server and the data on the master server.

[0048] LSN (Log Sequence Number): A unique identifier used in a database system to identify each log record in the transaction log. Typically, the LSN is an incrementing number used to record the order and position of each log entry.

[0049] This application primarily addresses database cluster scenarios based on master-slave replication, where there is not only one master server but also several slave servers. The master and slave servers store the same data and communicate and synchronize via a network to maintain data consistency and availability. To adapt to the current read-heavy, write-light business requirements, the master server typically handles write operations, while the slave servers handle read operations. However, due to synchronization delays or other factors between the master and slave servers, the data on the slave servers may not be up-to-date, leading to inconsistencies between query results and those on the master server, thus affecting the accuracy of the query results. For example, when a write operation is executed on the master server, the data on the slave server may not yet be fully synchronized. If a query operation is executed on the slave server at this time, the query results may reflect the old data state instead of the latest data state. This query inconsistency can lead to data inconsistency and incorrect business decisions.

[0050] To overcome the aforementioned technical shortcomings, related technologies propose setting master-slave replication as synchronous streaming replication. This method ensures real-time data synchronization between the master and slave servers; that is, after a write operation is completed on the master server, it must wait for the slave server to confirm receipt and apply the operation before proceeding to the next step. While this method guarantees data synchronization between the master and slave servers, synchronous streaming replication requires the master server to wait for confirmation from the slave server after a write operation, which increases the master server's response time. If the slave server's processing speed is slow or network latency is high, the master server may need to wait for a long time, thus affecting overall performance. Secondly, in synchronous streaming replication, there is a strong dependency between the master and slave servers. If the master server fails, the entire system may not function properly until the master server recovers or switches to a standby master server.

[0051] To overcome the aforementioned technical deficiencies, this application provides a data processing method. When a data processing request is determined to be a data read request, multiple slave servers are first polled, and one of them is identified as the target slave server. A first log sequence number stored in the target slave server is obtained, and then a second log sequence number stored in the master server is obtained. The synchronization latency difference between the master server and the target slave server is determined based on the first and second log sequence numbers. When the synchronization latency difference is determined to be less than or equal to a preset threshold and the number of polls is less than or equal to a preset maximum number of polls, it indicates that the latency difference between the data information stored in the target slave server determined in this poll and the data stored in the master server is small or non-existent. Polling is then stopped, and the data read request is assigned to the target slave server. The query result obtained by the target server based on the data read request is returned. This query result is consistent with the query result stored in the master server, ensuring the accuracy of the query result. Furthermore, this application can dynamically adjust the threshold corresponding to the synchronization latency difference according to different scenario requirements to meet the needs of different business scenarios.

[0052] It is understood that the data processing method and apparatus provided in this application can be embedded in the proxy layer of a database cluster architecture, providing more flexible query consistency control for businesses. First, this method introduces the concept of a consistency tolerance factor based on the synchronization latency difference between the latest log sequence numbers of the master and slave servers. This parameter can be dynamically adjusted according to business needs to meet the query consistency strength requirements of different business scenarios. Simultaneously, based on the existing proxy load balancing round-robin algorithm, this invention introduces the concept of a maximum number of consistency polls. Under the condition of meeting the business query consistency strength, its size can be dynamically adjusted according to business needs to meet the real-time query requirements of different business scenarios. Based on the consistency tolerance factor and the maximum number of consistency polls, this application provides a data processing method and apparatus that can ensure a good balance between business query real-time performance and query consistency performance, reasonably allocate query requests to the database cluster, and ensure that a request server that meets the query consistency strength is found.

[0053] It is understandable that a typical distributed database cluster includes multiple server nodes, each of which includes a master server and multiple slave servers. The data read and write logic of each server node is the same. This application describes the data processing method for a single server node.

[0054] The following is combined Figures 1-2 The first embodiment describes the data processing method of the present invention.

[0055] Figure 1 This invention provides one of the schematic flowcharts of a data processing method. Figure 2This is a second schematic diagram of a data processing method provided for an embodiment of the present invention. Please refer to it. Figure 1 and Figure 2 As shown, the data processing method includes:

[0056] S101. In response to the received data processing request, when it is determined that the data processing request is a data reading request, the following steps are executed cyclically.

[0057] In one embodiment, the proxy layer of the database cluster receives a data processing request from a client. This data processing request is typically in the form of an SQL statement. The SQL statement is then parsed to determine the type of the current data processing request. Common types of current data processing requests include data read requests and data write requests. When the type of the current data processing request is determined to be a data read request, the following steps S102, S103, and S104 are executed repeatedly to complete the data read request.

[0058] In one embodiment, in response to a received data processing request, when the data processing request is determined to be a data write request, the data write request is assigned to the master server, and a second log sequence number stored in the master server is returned.

[0059] S102, Polling step: Poll the multiple slave servers and determine one of them as the target slave server, and obtain the first log sequence number stored in the target slave server determined this time.

[0060] In load balancing, the round-robin algorithm plays a crucial role. It ensures that each server can participate in request processing, thus achieving load balancing. Its working principle is to distribute requests to each slave server sequentially according to a predefined order, and then repeat this process cyclically. Therefore, this invention utilizes the round-robin algorithm to select a specific slave server S. cur This will be from server S cur The target slave server selected in this polling is S, and the target slave server S is obtained. cur The first log sequence number is returned to the proxy layer.

[0061] The first log sequence number is the latest replay log sequence number of the target server, determined by the polling algorithm.

[0062] Furthermore, in practical application scenarios, in order to obtain a certain slave server S cur The latest replay log sequence number, one possible implementation is: when the proxy layer polls for S cur At that time, the database cluster agent distributes a query statement to S cur: select pg_last_wal_replay_lsn(); (for PostgreSQL) Query the latest log sequence number replayed from the server. And return to the proxy layer.

[0063] S103, Synchronization delay difference determination step: Obtain the second log sequence number stored in the master server, and determine the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number.

[0064] In one embodiment, the synchronization between the master server and the slave server is determined by the following formula.

[0065] Step delay difference:

[0066]

[0067] in, The synchronization latency difference between the master server and the slave server, M LSN The second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

[0068] The second log sequence number is the latest replay log sequence number of the master server.

[0069] In practical applications, the proxy layer uses... This is used to obtain the target determined by the current polling from server S. cur The synchronization latency difference with the master server, when The larger the value, the higher the data density between the master server and the target slave server S. cur The greater the difference in the data, the more... When the value is 0, it indicates that the target is from server S. cur It has the same data as the main server M.

[0070] S104, Polling Stop Determination Step: Determine whether the synchronization delay difference is less than or equal to a preset threshold and the number of polls is less than or equal to a preset maximum number of polls. If so, stop polling and allocate the data reading request to the target slave server, and return the query result obtained by the target server based on the data reading request.

[0071] In one embodiment, a threshold V for the synchronization latency difference can be set based on the degree of information consistency requirements between the master server and slave servers in the current application scenario. The higher the degree of information consistency requirements, the smaller the threshold is set, and the lower the degree of information consistency requirements, the larger the threshold is set. This threshold V is also called the consistency tolerance factor (hereinafter referred to as the consistency tolerance factor for convenience), and the consistency tolerance factor can also be understood as a consistency strength factor.

[0072] In one embodiment, a maximum number of polling iterations T can be preset, where the number of polling iterations t represents the real-time nature of the current query, and the proxy layer is based on the synchronization latency difference. And the number of polling iterations t can achieve a balance between query consistency and real-time performance.

[0073] For example, in scenarios with high real-time requirements for queries, the maximum number of polling iterations T can be set to a smaller value; conversely, in scenarios with high data consistency requirements, the synchronization latency can be increased. Set it to a smaller size.

[0074] In one embodiment, when the synchronization delay difference is less than or equal to a preset threshold and the number of polling iterations is less than or equal to a preset maximum number of polling iterations, it is determined that both the data consistency requirement and the real-time requirement of the current query meet the preset requirements. Therefore, polling stops, and the current data read request is assigned to the target slave server determined in this poll, returning the query result obtained by that target server based on the data read request. In other words, if And t≤T, indicating that the current slave server S cur While satisfying the real-time requirements of queries (t≤T), the strength of query consistency also meets the set consistency tolerance standard. Therefore, read operation requests will be distributed to slave server S. cur Return the query results and end the algorithm process.

[0075] S105. If not, increase the number of polling cycles and repeat the above polling steps, synchronization delay difference determination steps, and polling stop determination steps until polling stops.

[0076] In one embodiment, due to the latency of data transmission between the master server and the slave server, the synchronization latency difference found in the first few polls may not meet the requirements of the preset consistency tolerance factor. However, if the number of polls has not reached the maximum number of polls T (i.e., under the premise that the real-time requirement is met), the polling continues until both the synchronization latency difference and the number of polls meet the requirements, and then the query result is obtained from the target slave server determined in this instance.

[0077] In one embodiment, if it is determined whether the synchronization delay difference is greater than a preset threshold and the number of polling iterations equals a preset maximum number of polling iterations, then if so, polling is stopped and the data read request is allocated to the master server; the query result obtained by the master server based on the data read request is returned. In other words, if And t = T, meaning that under the condition of ensuring the real-time performance of the query (t ≤ T), the current target is from server S. cur The query consistency strength does not meet the set consistency tolerance standard. Furthermore, this polling is the last polling opportunity (t = T). Therefore, in order to meet the business's real-time requirements (T), this embodiment provides a better solution: directly distribute the read operation statement to the main server M, return the query results, and end the algorithm process. This also ensures the consistency and real-time requirements of data query.

[0078] As can be seen, the data processing method provided in this application, based on the synchronization latency difference of the latest log sequence numbers between the master server and the slave server, introduces a consistency tolerance factor, which can dynamically adjust the query consistency strength to meet the query consistency requirements of different business scenarios. Furthermore, based on existing load balancing algorithms, a maximum round-robin count design is introduced. While meeting the business query consistency strength requirements, its size can be dynamically adjusted according to business needs to satisfy the real-time query requirements of different business scenarios.

[0079] Furthermore, the data processing method provided in this application does not rely on additional middleware in terms of hardware and can be easily embedded into the database cluster proxy layer. The method is simple and efficient.

[0080] To more clearly describe the technical solution of this application, the following second embodiment is also provided.

[0081] Generally, to ensure high availability, scalability, and high concurrency of databases, many businesses adopt a master-slave database cluster model. This involves setting up a master database and several slave databases, synchronizing data from the master database to the slave databases. The master database handles writes, and the slave databases handle reads, thus distributing the database request pressure. Furthermore, if the master server fails, a slave server can be promoted to master to ensure service availability. However, during peak business periods, frequent SQL write operations and frequent changes to the data in the master database increase the amount of data synchronized from the master server to the slave servers. This leads to synchronization delays between the master and slave servers, causing discrepancies between the data queried from the master and slave servers, thus affecting the accuracy of query results. This invention provides an optimization method for query consistency in a database cluster. Please refer to [reference needed]. Figure 2 As shown, the master server is initialized as M, and the set of slave servers is C = {S1,...,S}. i ,S i+1 ,...,S N}, where N represents the number of slave server nodes.

[0082] In this embodiment, the technician first sets two custom parameters: a consistency tolerance factor V and a maximum number of polling iterations T. The consistency tolerance factor V (0 ≤ V ≤ M) LSNV: This parameter represents the tolerance level for master-slave query consistency. The larger V is, the lower the required query consistency strength. When V is 0, it indicates that strong consistency is required. The maximum number of consistency polls is T (T≥1), which represents the maximum number of polls required to find a slave server in the slave server set C that meets the consistency tolerance factor condition. The consistency polling count is initialized to t=1 (1≤t≤T).

[0083] In practical applications, different business scenarios may have varying requirements for query consistency and real-time performance. This depends on the nature of the business, the importance of the data, and the user's expectations of the data. Some businesses have very high requirements for query consistency, such as finance and e-commerce, which need to ensure data accuracy and consistency, but may not have high requirements for real-time performance. Some businesses have very high requirements for query real-time performance, such as logistics tracking systems or real-time monitoring systems, which need to display location, status, and other information in real time, but may not have high requirements for query accuracy. Therefore, when executing SQL queries, a trade-off between the two needs must be considered. In this embodiment, the consistency tolerance factor V is mainly set for the business's requirements for query consistency; the larger V is, the lower the required query consistency strength. When V is 0, it indicates that strong consistency is required. The maximum number of consistency polls T is mainly set for the business's requirements for real-time performance; the larger T is, the lower the real-time requirements of the business; when T is set to 1, it indicates that the business has extremely high real-time requirements and needs to return query results immediately. Through the two parameters introduced in this invention, technicians can weigh the relationship between real-time performance and consistency according to specific needs and business scenarios. For example, suppose a database cluster has 1 master and 4 slaves, and empirical testing shows that the maximum synchronization latency between master and slaves is typically around 10 MB. By setting several sets of parameters (V,T) = (10,1), (8,2), (6,3), (4,4), (2,5), and (0,6), it can be inferred that as the business logic is set to these sets of parameters sequentially, the requirements for query consistency increase sequentially, while the requirements for real-time performance decrease sequentially. For example, the parameter set (10,1) can tolerate query inconsistencies to a large extent but has extremely high real-time requirements; the parameter set (0,6) requires strong query consistency but has very lenient real-time requirements.

[0084] In addition, introducing the maximum number of consistent polling T not only allows for dynamic adjustment of the business's requirements for query real-time performance, but also ensures that the query consistency strength requirements are met while matching the original design intent of the database cluster's "master-write, slave-read" architecture as much as possible, thereby improving the concurrency capability of the database cluster.

[0085] The specific implementation steps of the data processing method provided in this embodiment are as follows:

[0086] Step 201: The database cluster agent accepts the data processing request sent by the client and parses the type of the data processing request.

[0087] In practical applications, a master-slave database cluster can be created in several ways. For example, some open-source databases can utilize their native clustering modes. In a MySQL cluster, the master server asynchronously sends binlog logs to the slave servers, which then receive and replay these logs using I / O and SQL threads, achieving master-slave data synchronization. In a PostgreSQL cluster, based on streaming replication, the master server asynchronously starts a Walsender process to send Wal logs to the slave servers, which receive and replay these logs using Walreciever and Startup processes. Alternatively, middleware can be used to create a cluster with different databases. For instance, the master server could be an Oracle database, and the slave servers could be several MySQL databases. Oracle's message synchronization mechanism and Kafka message queue technology can be used to encapsulate changes in the Oracle master database into fixed-format logs, which are then sent to Kafka as messages. The slave servers, acting as Kafka consumers, receive these messages, decapsulate the changes, and synchronize them to the slave servers. This example uses a PostgreSQL cluster to illustrate the subsequent implementation steps.

[0088] Database cluster proxies typically function as the upper layer of a database cluster, acting as intermediaries for the lower-level database cluster. They receive and distribute SQL statements from clients or applications, and often include features such as load balancing, failover, and query optimization. Using SQL keywords, the proxy can easily parse the type of the SQL statement. For example, the SELECT keyword usually indicates a read operation, while keywords like UPDATE, INSERT, or DELETE usually indicate a write operation. Common proxies include Pgpool and HAProxy. It's worth noting that this embodiment does not impose any requirements on the choice of proxy.

[0089] Step 202: Perform the following corresponding processing method according to the type of data processing request.

[0090] Step 2021: If the data processing request type is a write operation statement, dispatch the request to the master server M and return the latest log sequence number in M ​​(denoted as M). LSN The algorithm flows to the proxy layer and ends.

[0091] In practical applications, the master-slave read rule in a database cluster should be followed. When the proxy parses the statement as a write operation, it distributes the request to the master server. It's worth noting here that in order to return the latest log sequence number in M ​​(denoted as M...),...LSN In practical applications, a feasible implementation for the proxy layer is as follows: whenever the proxy parses an SQL statement as a write operation statement, the database cluster proxy appends a query statement to the write operation statement: `select sent_lsn from pg_stat_replication;` (for PostgreSQL), retrieving the latest log sequence number M sent from the primary database to the standby database. LSB The result is then returned to the proxy layer. The proxy layer can record this return result in a preset memory area or a cache such as Redis, ensuring that the latest log sequence number sent from the master database to the slave database can be stored and updated in a timely manner after each write operation.

[0092] Step 2022: If the data processing request type is a read operation statement, based on the existing load balancing round-robin algorithm of the database proxy, poll the set of slave servers C and obtain the target slave server S. cur The latest replay log sequence number (denoted as) );

[0093] In load balancing, the round-robin algorithm plays a crucial role. It ensures that each server can participate in request processing, thus achieving load balancing. Its working principle is to distribute requests to each slave server sequentially according to a predefined order, and then repeat this process cyclically. Therefore, this embodiment utilizes the round-robin algorithm to select a specific slave server S. cur For the target server, obtain the target server S. cur The latest replay log sequence number (i.e., the first log sequence number) is returned to the proxy layer.

[0094] Furthermore, in practical applications, in order to obtain a certain slave server S cur The latest replay log sequence number, one possible implementation is: when the proxy layer polls for S cur At that time, the database cluster agent distributes a query statement to S cur : select pg_last_wal_replay_lsn(); (for PostgreSQL) Query the latest log sequence number replayed from the server. And return to the proxy layer.

[0095] Step 203: Calculate M and S cur The difference in synchronization delay between them

[0096] In practical applications, the proxy layer uses... This is used to obtain the target determined by the current polling from server S. cur The synchronization latency difference with the master server, when The larger it is, the greater the difference between the data of the master server and that of the slave server S cur When is 0, it means that the slave server S cur has the same data as the master server M.

[0097] Step 204: Based on the preset consistency tolerance factor V and the maximum consistency polling times T, determine whether the current polling meets the stop polling condition. Specifically, it includes:

[0098] Step 2041: If and t ≤ T, stop polling, distribute the read operation request to the target slave server S cur , return the query result and end the algorithm process. It can be understood that if and t ≤ T, it means that while the current target slave server S cur meets the query real-time requirement (t ≤ T), the strength of its query consistency also meets the set consistency tolerance standard Therefore, the read operation request is distributed to the target slave server S cur , return the query result and end the algorithm process.

[0099] Step 2042: If and t < T, then t = t + 1 and go to Step 2022; it can be understood that if and t < T, it means that while ensuring the query real-time requirement is met (t ≤ T), the strength of the query consistency of the current target slave server S cur does not meet the set consistency tolerance standard Since t ≤ T, there is still a chance to continue polling the next target slave server S cur+1 , and repeat the above Steps 201 to Step 2041.

[0100] Step 2043: If and t = T, directly distribute the data reading request to the master server M, return the query result and end the algorithm process. It can be understood that if and t = T, it means that while ensuring the query real-time requirement is met (t ≤ T), the strength of the query consistency of the current target slave server S cur does not meet the set consistency tolerance standard And this polling is the last polling opportunity (t = T). Therefore, in order to meet the business requirement for real-time (T), a better solution is to directly distribute the read operation statement to the master server M, return the query result and end the algorithm process.

[0101] In practical applications, the proxy layer is based on Including the polling count t, and introducing a consistency tolerance factor V and a maximum consistency polling count T, a balance can be achieved between query consistency and real-time performance. Steps 2041, 2042, and 2043 described above can all be implemented in the proxy layer.

[0102] Furthermore, in this embodiment, a server polling algorithm involved in the above steps can be provided, the steps of which are as follows:

[0103] S1, randomly selects the server index value i = random(1,N) using a random function;

[0104] S2, from S i％N Polling begins sequentially, i.e., S i％N S (i+1)％N The polling continues until, after the polling count t = T, a slave server satisfying the query consistency strength (i.e., the consistency strength factor proposed in this invention) is not found, at which point the polling ends. Note that this invention is not limited to the above polling algorithm.

[0105] As can be seen, the data processing method provided in this embodiment introduces a consistency tolerance factor based on the synchronization latency difference of the latest log sequence numbers between the master server and the slave server. This allows for dynamic adjustment of query consistency strength, meeting the query consistency requirements of different business scenarios. Furthermore, based on existing load balancing algorithms, a maximum round-robin count is designed. While meeting the business query consistency strength requirements, this count can be dynamically adjusted according to business needs, satisfying the real-time query requirements of different business scenarios.

[0106] The data processing apparatus provided by the present invention will be described below. The data processing apparatus described below can be referred to in correspondence with the data processing method described above.

[0107] Figure 3 This is a schematic diagram of a data processing device structure provided in an embodiment of the present invention. Please refer to it. Figure 3 As shown, the data processing device includes: a request receiving unit 301, a polling unit 302, a synchronization delay difference determination unit 303, a polling stop determination unit 304, and a polling control unit 305.

[0108] The request receiving unit 301 is used to respond to the received data processing request and, when determining that the data processing request is a data reading request, trigger the polling unit 302, the synchronization delay difference determination unit 303, the polling stop determination unit 304, and the polling control unit 305 to work.

[0109] The polling unit 302 is used to poll the plurality of slave servers and determine one of them as the target slave server, and obtain the first log sequence number stored in the target slave server determined this time.

[0110] The synchronization delay difference determination unit 303 is used to obtain the second log sequence number stored in the master server and determine the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number.

[0111] The polling stop determination unit 304 is used to determine whether the synchronization delay difference is less than or equal to a preset threshold and whether the number of polling is less than or equal to a preset maximum number of polling. If so, the polling is stopped and the data reading request is allocated to the target slave server, and the query result obtained by the target server according to the data reading request is returned.

[0112] The polling control unit 305 is used to control the polling unit, the synchronization delay difference determination unit, and the polling stop determination unit to work in a loop until the polling stops when the number of polling is less than the preset maximum number of polling and the synchronization delay difference is greater than the preset threshold.

[0113] In one embodiment, the polling stop determination unit 304 is further configured to:

[0114] If the synchronization delay difference is greater than a preset threshold and the number of polling is equal to the preset maximum number of polling, then polling is stopped and the data read request is assigned to the master server; the query result obtained by the master server based on the data read request is returned.

[0115] In one embodiment, the request receiving unit 301 is further configured to:

[0116] In response to a received data processing request, if the data processing request is determined to be a data write request, the data write request is assigned to the master server, and the second log sequence number stored in the master server is returned.

[0117] In one embodiment, the synchronization delay difference determination unit 303 is specifically used for:

[0118] The synchronization latency difference between the master server and the slave server is determined by the following formula:

[0119]

[0120] in, The synchronization latency difference between the master server and the slave server, M LSN The second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

[0121] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions stored in the memory 430 to execute the aforementioned data processing method.

[0122] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0123] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to perform the data processing methods provided by the above methods.

[0124] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data processing methods provided by the methods described above.

[0125] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0126] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method applied in a distributed database cluster, the distributed database cluster comprising multiple server nodes, each server node comprising a master server and multiple slave servers; characterized in that, For any of the server nodes, the data processing method includes: In response to a received data processing request, when the data processing request is determined to be a data read request, the following steps are executed in a loop; The polling step involves polling the multiple slave servers and determining one of them as the target slave server, and obtaining the first log sequence number stored in the target slave server determined this time. The synchronization delay difference determination step involves obtaining the second log sequence number stored in the master server, and determining the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number; the synchronization delay difference is the difference between the second log sequence number and the first log sequence number. The polling stop determination step determines whether the synchronization delay difference is less than or equal to a preset threshold and whether the number of polls is less than or equal to a preset maximum number of polls. If so, the polling is stopped and the data reading request is allocated to the target slave server, and the query result obtained by the target slave server according to the data reading request is returned. If not, increase the number of polling iterations and repeat the above polling steps, synchronization delay difference determination steps, and polling stop determination steps until polling stops; The polling step also includes a server polling algorithm: S1, randomly selects the server index value using a random function. ; S2, from Polling begins sequentially, i.e. , Until the polling count If no slave server that meets the consistency strength factor is found, the polling ends; where T is the maximum number of consistency polls, the larger T is, the lower the real-time requirement of the business, and when T is set to 1, it means that the real-time requirement of the business is extremely high and the query result needs to be returned immediately; N is the number of slave servers; The preset threshold is a preset consistency tolerance factor V, which represents the degree tolerable for master-slave query consistency. The larger V is, the lower the required query consistency strength. When V is 0, it means that strong consistency is required.

2. The data processing method according to claim 1, characterized in that, The polling stop determination step also includes: If the synchronization delay difference is greater than a preset threshold and the number of polling is equal to the preset maximum number of polling, then polling is stopped and the data read request is assigned to the master server; the query result obtained by the master server based on the data read request is returned.

3. The data processing method according to claim 1 or 2, characterized in that, Also includes: In response to a received data processing request, if the data processing request is determined to be a data write request, the data write request is assigned to the master server, and the second log sequence number stored in the master server is returned.

4. The data processing method according to claim 1 or 2, characterized in that, The step of determining the synchronization latency difference between the master server and the target slave server based on the first log sequence number and the second log sequence number includes: The synchronization latency difference between the master server and the slave server is determined by the following formula: ； in, The synchronization latency difference between the master server and the slave server The second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

5. A data processing apparatus, applied in a distributed database cluster, the distributed database cluster comprising multiple server nodes, each server node comprising a master server and multiple slave servers; characterized in that, The data processing device includes: a request receiving unit, a polling unit, a synchronization delay difference determination unit, a polling stop determination unit, and a polling control unit; The request receiving unit is used to respond to the received data processing request and, when determining that the data processing request is a data reading request, trigger the polling unit, the synchronization delay difference determination unit, the polling stop determination unit, and the polling control unit to work. The polling unit is used to poll the plurality of slave servers and determine one of the slave servers as the target slave server, and obtain the first log sequence number stored in the target slave server determined this time. A synchronization delay difference determination unit is used to obtain a second log sequence number stored in the master server, and determine the synchronization delay difference between the master server and the target slave server based on the first log sequence number and the second log sequence number; the synchronization delay difference is the difference between the second log sequence number and the first log sequence number. The polling stop determination unit is used to determine whether the synchronization delay difference is less than or equal to a preset threshold and whether the number of polling is less than or equal to a preset maximum number of polling. If so, the polling is stopped and the data reading request is allocated to the target slave server, and the query result obtained by the target slave server according to the data reading request is returned. The polling control unit is used to control the polling unit, the synchronization delay difference determination unit, and the polling stop determination unit to work in a loop until polling stops when the number of polling is less than the preset maximum number of polling and the synchronization delay difference is greater than the preset threshold. The polling unit also has a server polling algorithm function, including: S1, randomly selects the server index value using a random function. ; S2, from Polling begins sequentially, i.e. , Until the polling count If no slave server that meets the consistency strength factor is found, the polling ends; where T is the maximum number of consistency polls, the larger T is, the lower the real-time requirement of the business, and when T is set to 1, it means that the real-time requirement of the business is extremely high and the query result needs to be returned immediately; N is the number of slave servers; The preset threshold is a preset consistency tolerance factor V, which represents the degree tolerable for master-slave query consistency. The larger V is, the lower the required query consistency strength. When V is 0, it means that strong consistency is required.

6. The data processing apparatus according to claim 5, characterized in that, The polling stop determination unit is also used for: If the synchronization delay difference is greater than a preset threshold and the number of polling is equal to the preset maximum number of polling, then polling is stopped and the data read request is allocated to the master server. Return the query results obtained by the main server based on the data read request.

7. The data processing apparatus according to claim 5 or 6, characterized in that, The request receiving unit is further configured to respond to the received data processing request, and when determining that the data processing request is a data write request, allocate the data write request to the master server and return the second log sequence number stored in the master server.

8. The data processing apparatus according to claim 5 or 6, characterized in that, The synchronization delay difference determination unit is specifically used for: The synchronization latency difference between the master server and the slave server is determined by the following formula: ； in, The synchronization latency difference between the master server and the slave server The second log sequence number stored in the primary server; The first log sequence number stored from the server for the target.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 4.

Images