A data processing method for a database cluster
By setting database transaction IDs and table transaction IDs for the slave databases in the database cluster, and combining different strategies to select the target database, the problem of low data reading accuracy caused by different synchronization speeds of slave databases is solved, achieving more efficient and accurate data reading.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI KEZHONG HENGSHENG CLOUD COMPUTING TECH CO LTD
- Filing Date
- 2022-10-11
- Publication Date
- 2026-05-29
AI Technical Summary
In a database cluster, different synchronization speeds between databases can lead to low data read accuracy, especially when reading data from a database with a slower synchronization speed, which may result in incomplete data.
By setting database transaction IDs and table transaction IDs for each slave database, and combining the first and second strategies, the target database is determined for data retrieval. The first strategy selects the target database based on the table transaction ID when the database transaction IDs are inconsistent; the second strategy selects the target database by considering the slave database's access volume and historical access data when the database transaction IDs are consistent.
It improved the success rate and efficiency of data retrieval, ensured that the retrieved data content was updated in a timely and comprehensive manner, and balanced the access volume from the database.
Smart Images
Figure CN116186153B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a data processing method for database clusters. Background Technology
[0002] Databases, as the infrastructure for storing data, are widely used in various products in the internet field. To improve the load capacity of these products, database clustering technology has been developed. Typically, a database cluster consists of one master database and multiple slave databases. The master database is responsible for writing data, while the slave databases are responsible for reading data. Slave databases can synchronize data by backing up data from the master database and synchronously execute corresponding operations when the master database performs write operations.
[0003] During the data synchronization process between the master and slave databases, the synchronization speeds differ. If some slave databases fail to synchronize in a timely manner, discrepancies may occur between the different slave databases. Therefore, when reading data from a slave database with a slower synchronization speed, incomplete data may occur, leading to data read failures. Furthermore, the varying synchronization speeds of the slave databases result in lower data read accuracy. Summary of the Invention
[0004] This application provides a data processing method for a database cluster to solve the problem of low data reading accuracy caused by different synchronization speeds from the database.
[0005] In a first aspect, embodiments of this application provide a data processing method for a database cluster, the database cluster comprising: a master database and multiple slave databases; each slave database storing multiple data tables; each slave database associated with a database transaction ID; each data table associated with its own table transaction ID; the method comprising:
[0006] Upon receiving a target data read instruction, compare the database transaction IDs of each slave database;
[0007] If the database transaction IDs of the slave databases are inconsistent, the target database is determined from the slave databases according to the first strategy and the table transaction ID; otherwise, the target database is determined from the slave databases according to the second strategy.
[0008] Read the target data from the target database.
[0009] In one possible implementation, determining the target database from each slave database based on the first strategy and the table transaction ID includes:
[0010] The target table ID for storing the target data is determined according to the target data read instruction;
[0011] Find the table transaction ID of each target table in the slave database based on the target table ID;
[0012] Determine the target database as the slave database corresponding to the latest table transaction ID.
[0013] In one possible implementation, the method further includes:
[0014] When the latest table transaction ID corresponds to multiple slave databases, the target database is determined based on the current access data of the multiple slave databases.
[0015] In one possible implementation, determining the target database based on multiple currently accessed data from the database includes:
[0016] Determine the database with the fewest currently accessing IPs as the target database; or,
[0017] Determine the target database as the one with the fewest currently accessed data items; or,
[0018] The database with the shortest current access time is selected as the target database.
[0019] In one possible implementation, determining the target database from each slave database according to the second strategy includes:
[0020] The target database is determined based on the historical access data of each slave database.
[0021] In one possible implementation, determining the target database based on historical access data from each slave database includes:
[0022] Determine the database with the fewest historical accesses as the target database; or,
[0023] The target database is the one with the fewest historical access data items; or,
[0024] The database with the shortest cumulative historical access duration is selected as the target database.
[0025] In one possible implementation, the method further includes, before determining the target database based on the historical access data of each slave database, the method further includes:
[0026] Compare the transaction IDs in the tables of each slave database;
[0027] The operation of determining the target database based on the historical access data of each slave database is performed when the table transaction IDs are consistent.
[0028] In one possible implementation, the method further includes:
[0029] When the database transaction IDs of each slave database are consistent, but the table transaction IDs are inconsistent, the target table ID for storing the target data is determined according to the target data read instruction.
[0030] Find the table transaction ID of each target table in the slave database based on the target table ID;
[0031] Determine the target database as the slave database corresponding to the latest table transaction ID.
[0032] In one possible implementation, the method further includes:
[0033] When there is an anomaly in the database, a record item is generated according to the operation corresponding to the target data read instruction, and the record item is recorded in the fault table corresponding to the abnormal database.
[0034] In one possible implementation, the method further includes:
[0035] When the database recovers from an anomaly, each record in the fault table corresponding to the anomaly is written to the corresponding data table, and the data in the fault table is cleared.
[0036] In one possible implementation, before writing the exception from each record in the corresponding fault table of the database to the corresponding data table, the method further includes:
[0037] Determine the duration of database failures to identify anomalies;
[0038] When the fault duration is less than a set threshold, the operation of writing the abnormality from each record item in the target fault table in the database to the corresponding data table is performed.
[0039] Otherwise, back up the data from the primary database to the faulty slave database and clear the data in the faulty table.
[0040] In one possible implementation, determining the duration of the database failure due to the anomaly includes:
[0041] The duration of the target fault is determined based on the time information of the record entries in the corresponding fault table in the database.
[0042] Secondly, embodiments of this application provide a data processing apparatus for a database cluster, the database cluster comprising: a master database and multiple slave databases; each slave database storing multiple data tables; each slave database associated with a database transaction ID; each data table associated with its own table transaction ID; the apparatus comprising:
[0043] The comparison module is used to compare the database transaction IDs of each slave database when a target data read instruction is received;
[0044] The determination module is used to determine the target database from each slave database according to a first strategy and a table transaction ID when the database transaction IDs of each slave database are inconsistent; otherwise, it determines the target database from each slave database according to a second strategy.
[0045] A reading module is used to read the target data from the target database.
[0046] In one possible implementation, the determining module is specifically used to determine the target table ID storing the target data according to the target data reading instruction, find the table transaction ID of the target table in each slave database according to the target table ID, and determine the slave database corresponding to the latest table transaction ID as the target database.
[0047] In one possible implementation, the determining module is further configured to determine the target database based on the current access data of the multiple slave databases when the latest table transaction ID corresponds to multiple slave databases.
[0048] In one possible implementation, determining the target database based on multiple currently accessed data from the database includes:
[0049] Determine the database with the fewest currently accessing IPs as the target database; or,
[0050] Determine the target database as the one with the fewest currently accessed data items; or,
[0051] The database with the shortest current access time is selected as the target database.
[0052] In one possible implementation, the determining module is specifically used to determine the target database based on the historical access data of each slave database.
[0053] In one possible implementation, determining the target database based on historical access data from each slave database includes:
[0054] Determine the database with the fewest historical accesses as the target database; or,
[0055] The target database is the one with the fewest historical access data items; or,
[0056] The database with the shortest cumulative historical access duration is selected as the target database.
[0057] In one possible implementation, the device further includes:
[0058] The recording module is used to generate a record item according to the operation corresponding to the target data read instruction when there is an abnormality in the database, and record the record item in the fault table corresponding to the abnormal database.
[0059] In one possible implementation, the device further includes:
[0060] The recovery module is used to write each record item in the fault table corresponding to the faulty database to the corresponding data table and clear the data in the fault table when the database recovers from an anomaly.
[0061] In one possible implementation, the recovery module is further configured to determine the duration of the failure from the database; when the duration of the failure is less than a set threshold, to perform the operation of writing each record item in the target failure table corresponding to the database from the failure to the corresponding data table.
[0062] Otherwise, back up the data from the primary database to the faulty slave database and clear the data in the faulty table.
[0063] In one possible implementation, the recovery module is specifically used to determine the target fault duration based on the time information of the record item in the fault table corresponding to the anomaly in the database.
[0064] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect or any possible implementation of the first aspect above.
[0065] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect or any possible implementation of the first aspect.
[0066] This application provides a data processing method for a database cluster. Upon receiving a target data read instruction, it compares the database transaction IDs of each slave database and determines the synchronization level between each slave database and the master database based on the database transaction IDs. If the database transaction IDs of the slave databases are inconsistent, their synchronization levels differ. The target database is then determined from each slave database according to a first strategy and the table transaction ID, ensuring successful data retrieval from the data table. If the database transaction IDs of the slave databases are consistent, the target database is determined from each slave database according to a second strategy, ensuring balanced access volume across slave databases and improving data read efficiency. This application comprehensively considers both database transaction IDs and table transaction IDs when reading data according to the target data read instruction, ensuring the most timely updates to the corresponding data table content and improving the success rate and efficiency of data reads. Attached Figure Description
[0067] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0068] Figure 1 This is an application scenario diagram of a data processing method for a database cluster provided in an embodiment of this application;
[0069] Figure 2 This is a flowchart illustrating the implementation of a data processing method for a database cluster according to an embodiment of this application;
[0070] Figure 3 This is a schematic diagram of the structure of a data processing apparatus for a database cluster provided in an embodiment of this application;
[0071] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0072] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0073] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.
[0074] Unless otherwise stated, the term "multiple" means two or more.
[0075] In this embodiment, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.
[0076] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.
[0077] Figure 1This is an application scenario diagram of a data processing method for a database cluster provided in an embodiment of this application. The application scenario includes a terminal device, a master database, slave database 1, and slave database 2. There can be two or more slave databases; only two are shown as an example in the diagram. The master database includes multiple data tables. Each slave database is communicatively connected to the master database and maintains data synchronization with the master database through replication, while providing read-only services to the outside world.
[0078] The terminal device receives write operations from the user, such as inserting, modifying, or deleting data. Based on the user's request, the terminal device sends a write command to the main database, which carries the data to be written and a data type identifier.
[0079] After receiving a write command from the terminal device, the master database determines the target data table for the write operation based on the write data and data type identifier carried in the command, and then performs the corresponding write operation on that table, such as inserting, modifying, or deleting data. Additionally, the write command needs to be synchronously sent to the slave database so that the slave database can perform the corresponding write operation accordingly.
[0080] After receiving write commands synchronously from the master database, the slave database performs corresponding write operations on the data table to be updated, such as inserting, modifying, or deleting data. The slave database also handles database query tasks, receiving query commands from the terminal device, executing corresponding query operations, obtaining query results, and sending the results to the terminal device for display.
[0081] The write commands sent synchronously from different databases and the corresponding write operations executed at varying speeds can lead to synchronization delays in some databases. Consequently, data read from databases with slower synchronization speeds exhibits lower accuracy. This application aims to address the problem of low data read accuracy caused by varying synchronization speeds among databases.
[0082] To address this issue, this application requires determining the synchronization level between the data in each slave database and the master database when reading data from the slave databases. A higher synchronization level indicates more complete data, increasing the probability of successful data retrieval. To facilitate determining the synchronization level of each slave database, a database transaction ID is set for each slave database. This database transaction ID is accumulated or updated based on the number of operations performed on the slave database. Therefore, the age of each slave database can be determined based on the size or age of its database transaction ID.
[0083] In addition, given that there are multiple data tables in the database, and the synchronization level of the same data table varies between different databases, a table transaction ID is set for each data table to ensure that data can be successfully read from different data tables.
[0084] To facilitate understanding of database transaction IDs and table transaction IDs, Tables 1 and 2 are used in conjunction with specific embodiments for explanation.
[0085] Table 1 below is a list of transaction IDs from a database in a specific embodiment of this application.
[0086] Table 1
[0087] Serial Number type Transaction ID 1 Library transactions 3 2 Table X Transaction 1 3 Table Y Transaction 2
[0088] This example uses a database consisting of two tables, X and Y. A database transaction ID is set for the database, and a table transaction ID is set for each table. Therefore, in the case of tables X and Y, the table transaction ID includes both the transaction ID of table X and the transaction ID of table Y.
[0089] In addition, if the database transaction ID is an ID that facilitates global data management, then the database transaction ID is the sum of the table transaction IDs of all data tables.
[0090] Table 2 is a list of multiple transaction IDs from the database in a specific embodiment of this application.
[0091] Table 2
[0092]
[0093]
[0094] This example uses three databases: database A, database B, and database C. The number of tables in each database is the same; comparisons are made only using the transaction IDs of tables X, Y, and Z.
[0095] As can be seen from Table 2, the transaction ID of database A is 120, which is greater than the transaction IDs of databases B and C (i.e., 119). In this case, database A has the highest degree of data synchronization with the master database, meaning that the data in database A is more comprehensive.
[0096] Furthermore, given that there are multiple tables in the database, the synchronization level of each table varies even when the database transaction IDs are the same. As shown in Table 2, database B and database C have the same database transaction IDs, but tables X and Z have different table transaction IDs. Specifically, the synchronization level of tables X and Z in database B is lower than that in database C, meaning the data is not as complete. In subsequent tables (X, Y, Z) or other tables, the synchronization level of tables in database B will be higher than that of the corresponding tables in database C.
[0097] As can be further seen from Tables 1 and 2, different levels of backup or synchronization between the slave database and the master database will result in different data, and the data in different data tables will also be different. Therefore, when reading from the slave database, it is necessary to consider not only the synchronization level of the slave database, but also the synchronization level of the data tables.
[0098] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.
[0099] Figure 2 This is a flowchart illustrating the implementation of a data processing method for a database cluster according to another embodiment of this application, as shown below. Figure 2 As shown, the method includes the following steps:
[0100] S201. Upon receiving the target data read instruction, compare the database transaction IDs of each slave database.
[0101] In this application, the implementing entity is a control terminal or server that centrally manages each database. The target data read instruction is sent by the user through a terminal device. The target data read instruction must include at least the data type or storage location of the target data to be read, so that the control terminal or server can determine the data's storage location based on the instruction.
[0102] S202. When the database transaction IDs of the slave databases are inconsistent, the target database is determined from each slave database according to the first strategy and the table transaction ID; otherwise, the target database is determined from each slave database according to the second strategy.
[0103] As can be seen from the above, when the database transaction IDs of different slave databases are inconsistent, the degree of data synchronization between different slave databases is different. Specifically, the degree of synchronization of data tables is different. Therefore, in order to ensure that data can be read successfully, it is necessary to judge the degree of data synchronization of each data table based on the first strategy and the table transaction ID to improve the probability of successful data reading.
[0104] When all database transaction IDs are consistent across all databases, it is assumed that the data synchronization level between each database and the master database is consistent. At this point, in order to ensure that data can be read quickly, it is necessary to judge the operating pressure, access pressure, or data transmission performance of each database according to the second strategy in order to improve data reading efficiency.
[0105] S203. Read target data from the target database.
[0106] In this embodiment, upon receiving a target data read instruction, the database transaction IDs of each slave database are compared, and the synchronization level between each slave database and the master database is determined based on the database transaction IDs. If the database transaction IDs of the slave databases are inconsistent, their synchronization levels differ. The target database is then determined from each slave database according to a first strategy and the table transaction ID, ensuring successful data retrieval from the data table. If the database transaction IDs of the slave databases are consistent, the target database is determined from each slave database according to a second strategy, ensuring balanced access volume across slave databases and improving data read efficiency. This application comprehensively considers both database transaction IDs and table transaction IDs when reading data according to the target data read instruction, ensuring that the corresponding data table content is updated most promptly during data read, and improving the success rate and efficiency of data read.
[0107] In one possible implementation, step S202, determining the target database from each slave database based on the first strategy and the table transaction ID, includes:
[0108] Determine the target table ID for storing the target data based on the target data read instruction;
[0109] Find the table transaction ID of each target table in the slave database based on the target table ID;
[0110] Determine the target database as the slave database corresponding to the latest table transaction ID.
[0111] The target data read instruction includes the target data type and / or the table ID of the table to which the target data belongs. After obtaining the target data read instruction, the table ID of the target data to be read, i.e. the target table ID, can be further determined by parsing the target data read instruction. Then, the data read task can be executed from the corresponding table based on the target table ID.
[0112] This implementation primarily addresses the case where the latest table transaction ID is unique. In this case, the data table corresponding to the latest table transaction ID is the one with the fastest update speed and the most comprehensive data from the same table in the database, ensuring successful data retrieval.
[0113] Based on the aforementioned implementation methods, in one possible implementation, the method further includes:
[0114] When the latest table transaction ID corresponds to multiple slave databases, the target database is determined based on the current access data of the multiple slave databases.
[0115] This implementation primarily addresses the scenario where the latest table transaction ID is multiple. In this case, the data in the multiple tables is consistent, and the probability of successfully retrieving data is the same regardless of which table is accessed. The target database is determined based on the current access data from multiple slave databases. This allows for assessment of the communication pressure or operating speed of each slave database, enabling the selection of one to perform the data read operation and improving data retrieval efficiency.
[0116] In different embodiments, the method for determining the target database based on the current access data of multiple slave databases varies. Optionally, the slave database with the fewest currently accessing IPs can be determined as the target database; or, the slave database with the fewest currently accessed data items can be determined as the target database; or, the slave database with the shortest current access time can be determined as the target database.
[0117] The number of accessing IPs, the number of accessed data items, and the access time can be obtained based on the corresponding port of each slave database or recorded according to the signals of the corresponding port of each slave database.
[0118] In this implementation, when the number of currently accessing IPs is the fewest, the number of currently accessing data items is the fewest, or the current access time is the shortest, the pressure on the database is relatively low, the data throughput and response speed are high, and the data can be read quickly.
[0119] In one possible implementation, step S202, determining the target database from each slave database according to the second strategy, includes: determining the target database based on the historical access data of each slave database.
[0120] In different embodiments, the method for determining the target database based on the historical access data of each slave database varies. Optionally, this may include determining the slave database with the fewest historical accesses as the target database; or, determining the slave database with the fewest historical access data items as the target database; or, determining the slave database with the shortest cumulative historical access duration as the target database.
[0121] In this implementation, the longer the database is used, the more its performance will gradually decline, and the data reading rate will gradually slow down. Therefore, assuming that the data in each slave database is consistent, determining the slave database with the fewest historical accesses, the fewest historical access data items, or the shortest historical access duration as the target database can improve the efficiency of data reading. In addition, it can balance the performance of each slave database.
[0122] The above embodiments primarily consider the scenario where the database transaction IDs of all slave databases are consistent, and the synchronization level of the data tables in each slave database is also consistent. However, when there are a large number of data tables, it is inevitable that some data tables will have inconsistent update levels, while the database transaction IDs of all slave databases will be consistent.
[0123] Therefore, in another possible implementation, before determining the target database based on the historical access data of each slave database, the following steps are also included:
[0124] Compare the transaction IDs in the tables of each slave database;
[0125] When all table transaction IDs are consistent, the target database is determined based on the historical access data of each slave database.
[0126] Correspondingly, in another possible implementation, the method also includes:
[0127] When the database transaction IDs of each slave database are consistent, but the table transaction IDs are inconsistent, the target table ID for storing the target data is determined according to the target data read instruction.
[0128] Find the table transaction ID of each target table in the slave database based on the target table ID;
[0129] Determine the target database as the slave database corresponding to the latest table transaction ID.
[0130] In one possible implementation, the method further includes: when there is a database exception, generating a record item according to the operation corresponding to the target data read instruction, and recording the record item in the fault table corresponding to the database exception.
[0131] In this implementation, the fault table can be stored in the slave database, improving the relevance and accuracy of data storage. Alternatively, the fault table can be stored in the master database or a control terminal that centrally manages the slave databases, thereby enhancing the level of centralized management of the slave databases. Furthermore, establishing a corresponding fault table for each slave database can improve...
[0132] In one possible implementation, the method further includes:
[0133] When the database recovers from an anomaly, each record in the fault table corresponding to the anomaly is written to the corresponding data table, and the data in the fault table is cleared.
[0134] In this implementation, when a database fails, the operation items recorded in the fault table during the failure period can be used to achieve incremental recovery from the database. When recovering data from the slave database based on the fault table, the data synchronization rate between the slave and master databases is improved. Furthermore, the data transfer pressure on the master database during the recovery process is avoided.
[0135] In one possible implementation, before writing the exceptions from the records in the corresponding fault table in the database to the corresponding data table, the following steps are also included:
[0136] Determine the duration of database failures to identify anomalies;
[0137] When the fault duration is less than the set threshold, the operation of writing the abnormal from each record item in the target fault table in the database to the corresponding data table is executed.
[0138] Otherwise, back up the data from the primary database to the faulty slave database and clear the data in the faulty table.
[0139] In this implementation, during the period of database failure, data is copied from the master database to achieve full data recovery. This avoids changes to the master database during the period of database failure, ensures that the data in the slave database and the master database are completely synchronized, and improves the accuracy of data recovery.
[0140] In one possible implementation, determining the fault duration from the database includes: determining the target fault duration based on the time information of the record entries in the fault table corresponding to the fault from the database.
[0141] Optionally, the target fault duration can be determined based on the time information of the earliest record in the fault table and the current time; or, the target fault duration can be determined based on the time information of the earliest and latest records in the fault table.
[0142] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0143] The following are device embodiments of this application. For details not described in detail, please refer to the corresponding method embodiments described above.
[0144] Figure 3 This is a schematic diagram of the structure of a data processing device for a database cluster provided in an embodiment of this application, as shown below. Figure 3 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown, such as... Figure 3 As shown, the device includes: a comparison module 301, a determination module 302, and a reading module 303.
[0145] The comparison module 301 is used to compare the database transaction IDs of each slave database when a target data read instruction is received;
[0146] The determination module 302 is used to determine the target database from each slave database according to the first strategy and the table transaction ID when the database transaction IDs of each slave database are inconsistent; otherwise, it determines the target database from each slave database according to the second strategy.
[0147] The reading module 303 is used to read target data from the target database.
[0148] In one possible implementation, the determining module 302 is specifically used to determine the target table ID storing the target data according to the target data reading instruction, find the table transaction ID of the target table in each slave database according to the target table ID, and determine the slave database corresponding to the latest table transaction ID as the target database.
[0149] In one possible implementation, the determining module 302 is further configured to determine the target database based on the current access data of the multiple slave databases when the latest table transaction ID corresponds to multiple slave databases.
[0150] In one possible implementation, the target database is determined based on multiple currently accessed data from the database, including:
[0151] Determine the database with the fewest currently accessing IPs as the target database; or,
[0152] Determine the target database as the one with the fewest currently accessed data items; or,
[0153] The database with the shortest current access time is selected as the target database.
[0154] In one possible implementation, the determining module 302 is specifically used to determine the target database based on the historical access data of each slave database.
[0155] In one possible implementation, the target database is determined based on historical access data from each slave database, including:
[0156] Determine the database with the fewest historical accesses as the target database; or,
[0157] The target database is the one with the fewest historical access data items; or,
[0158] The database with the shortest cumulative historical access duration is selected as the target database.
[0159] In one possible implementation, the device further includes:
[0160] The logging module is used to generate a log entry based on the operation corresponding to the target data read instruction when there is a database exception, and to record the log entry in the fault table corresponding to the database exception.
[0161] In one possible implementation, the device further includes:
[0162] The recovery module is used to write each record item in the fault table corresponding to the faulty database to the corresponding data table and clear the data in the fault table when the database recovers from an anomaly.
[0163] In one possible implementation, the recovery module is also used to determine the duration of the failure from the database; when the duration of the failure is less than a set threshold, the module performs the operation of writing each record item in the target failure table corresponding to the failure from the database into the corresponding data table.
[0164] Otherwise, back up the data from the primary database to the faulty slave database and clear the data in the faulty table.
[0165] In one possible implementation, the recovery module is specifically used to determine the target fault duration based on the time information of the record entries in the corresponding fault table in the database.
[0166] In this embodiment, upon receiving a target data read instruction, the database transaction IDs of each slave database are compared, and the synchronization level between each slave database and the master database is determined based on the database transaction IDs. If the database transaction IDs of the slave databases are inconsistent, their synchronization levels differ. The target database is then determined from each slave database according to a first strategy and the table transaction ID, ensuring successful data retrieval from the data table. If the database transaction IDs of the slave databases are consistent, the target database is determined from each slave database according to a second strategy, ensuring balanced access volume across slave databases and improving data read efficiency. This application comprehensively considers both database transaction IDs and table transaction IDs when reading data according to the target data read instruction, ensuring that the corresponding data table content is updated most promptly during data read, and improving the success rate and efficiency of data read.
[0167] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 4 As shown, the electronic device 4 in this embodiment includes: a processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the processor 40. When the processor 40 executes the computer program 42, it implements the steps described in the various data processing method embodiments for database clusters, for example... Figure 2 Steps S201 to S203 are shown. Alternatively, when the processor 40 executes the computer program 42, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 3 The functions of modules 301 to 303 are shown.
[0168] For example, the computer program 42 can be divided into one or more modules / units, which are stored in the memory 41 and executed by the processor 40 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 42 in the electronic device 4. For example, the computer program 42 can be divided into... Figure 3 Modules 301 to 303 are shown.
[0169] The electronic device 4 can be a desktop computer, laptop, handheld computer, or cloud server, etc. The electronic device 4 may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.
[0170] The processor 40 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.
[0171] The memory 41 can be an internal storage unit of the electronic device 4, such as a hard disk or memory. The memory 41 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 41 can include both internal and external storage units of the electronic device 4. The memory 41 is used to store the computer program and other programs and data required by the electronic device. The memory 41 can also be used to temporarily store data that has been output or will be output.
[0172] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0173] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0174] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0175] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.
[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0178] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various data processing method embodiments for database clusters described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0179] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A data processing method for a database cluster, characterized in that, The database cluster includes: a master database and multiple slave databases; each slave database stores multiple data tables; each slave database is associated with a database transaction ID; each data table is associated with its own table transaction ID; the method includes: Upon receiving a target data read instruction, compare the database transaction IDs of each slave database; If the database transaction IDs of the slave databases are inconsistent, the target database is determined from the slave databases according to the first strategy and the table transaction ID; otherwise, the target database is determined from the slave databases according to the second strategy. Read the target data from the target database; The step of determining the target database from each slave database based on the first strategy and the table transaction ID includes: The target table ID for storing the target data is determined according to the target data read instruction; Find the table transaction ID of each target table in the slave database based on the target table ID; Determine the target database as the slave database corresponding to the latest table transaction ID.
2. The method according to claim 1, characterized in that, Also includes: When the latest table transaction ID corresponds to multiple slave databases, the target database is determined based on the current access data of the multiple slave databases.
3. The method according to claim 2, characterized in that, The step of determining the target database based on multiple currently accessed data from the database includes: Determine the database with the fewest currently accessing IPs as the target database; or, Determine the target database as the one with the fewest currently accessed data items; or, The database with the shortest current access time is selected as the target database.
4. The method according to claim 1, characterized in that, The step of determining the target database from each slave database according to the second strategy includes: The target database is determined based on the historical access data of each slave database.
5. The method according to claim 4, characterized in that, The step of determining the target database based on the historical access data of each slave database includes: Determine the database with the fewest historical accesses as the target database; or, The target database is the one with the fewest historical access data items; or, The database with the shortest cumulative historical access duration is selected as the target database.
6. The method according to any one of claims 1 to 5, characterized in that, Also includes: When there is an anomaly in the database, a record item is generated according to the operation corresponding to the target data read instruction, and the record item is recorded in the fault table corresponding to the abnormal database.
7. The method according to claim 6, characterized in that, Also includes: When the database recovers from an anomaly, each record in the fault table corresponding to the anomaly is written to the corresponding data table, and the data in the fault table is cleared.
8. The method according to claim 7, characterized in that, Before writing the exceptions from the corresponding records in the fault table of the database to the corresponding data table, the process also includes: Determine the duration of database failures to identify anomalies; When the fault duration is less than a set threshold, the operation of writing the abnormality from each record item in the target fault table in the database to the corresponding data table is performed. Otherwise, back up the data from the primary database to the faulty slave database and clear the data in the faulty table.
9. The method according to claim 8, characterized in that, The determination of the database failure duration includes: The duration of the target fault is determined based on the time information of the record entries in the corresponding fault table in the database.