Database data processing method and apparatus, device, and storage medium

Abstract

The application provides a database data processing method, device and equipment and a storage medium. The application can be applied to the field of cloud computing. The method comprises the following steps: performing incremental change detection on a master database to identify incremental data that needs to be backed up; predicting a backup starting time of the incremental data based on the amount of historical transaction processing data and the amount of incremental data of the master database; performing backup on the incremental data according to the predicted backup starting time, and restoring the backed-up incremental data to a target delayed reading time by using a delay database in the case that the backup is completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud computing, and in particular to a database data processing method, device, equipment and storage medium. BACKGROUND

[0002] In the management system of a distributed database, the data of the master database needs to be periodically backed up. With the exponential growth of the amount of data stored in a distributed manner, a delayed database architecture is usually used to synchronize the master database data in real time and provide a delayed read function.

[0003] In the process of implementing the present application, at least the following problems exist in the related art. When processing large-scale data, real-time synchronization of a delayed database consumes a large amount of system resources, resulting in a long synchronization process, which not only affects the running efficiency of the master database, but also reduces the speed of data recovery, ultimately leading to a decrease in overall resource utilization. SUMMARY

[0004] Therefore, the present application provides a database data processing method, device, equipment and storage medium.

[0005] One aspect of the present application provides a database data processing method, comprising: performing incremental change detection on a master database to identify incremental data that needs to be backed up; predicting a backup start time of the incremental data based on the amount of historical transaction processing data of the master database and the amount of the incremental data; performing backup on the incremental data according to the predicted backup start time, and restoring the backed up incremental data to a target delayed read time using a delayed database if the backup is completed.

[0006] According to an embodiment of the present application, the incremental change detection on the master database to identify the incremental data that needs to be backed up comprises: obtaining a change bitmap of the master database, wherein the master database records the block number of the data page that is modified in a transaction in the change bitmap during the transaction commit phase; loading the corresponding data page from the storage unit of the master database based on the plurality of block numbers recorded in the change bitmap to obtain the incremental data.

[0007] According to an embodiment of the present application, the method further comprises: detecting the number of block numbers recorded in the change bitmap; maintaining a polling waiting state until the number of recorded block numbers increases to a preset threshold value if it is determined that the number of block numbers is less than the preset threshold value; and performing the operation of loading the corresponding data page from the storage unit of the master database based on the plurality of block numbers recorded in the change bitmap if it is determined that the number is greater than or equal to the preset threshold value.

[0008] According to the embodiment of the present application, the method for predicting the backup starting time of the incremental data based on the historical transaction processing data and the data volume of the incremental data comprises: determining an estimated backup duration according to the historical backup rate of the master database and the data volume of the incremental data; counting the transaction processing volume in the historical transaction processing data to generate a transaction processing rate curve; extracting the trend change feature of the transaction processing volume by analyzing the transaction processing rate curve; and predicting the backup starting time of the incremental data according to the trend change feature and the estimated backup duration, wherein the backup ending time determined based on the backup starting time and the estimated backup duration is earlier than the target delayed reading time.

[0009] According to the embodiment of the present application, the method for predicting the backup starting time of the incremental data according to the trend change feature and the estimated backup duration comprises: identifying a plurality of candidate time intervals in which the transaction processing volume is lower than a preset number from the time processing sequence of the transaction processing curve based on the trend change feature; determining a target time interval which is earlier than the target delayed reading time and closest to the target delayed reading time from the plurality of candidate time intervals; and inversely calculating the backup starting time based on the end boundary time of the target time interval and the estimated backup duration.

[0010] According to the embodiment of the present application, the method further comprises: creating a data consistency point based on the backup ending time in the case of backup completion; and storing the data consistency point and the transaction log corresponding to the incremental data of the master database in association.

[0011] According to the embodiment of the present application, the method for restoring the incremental data to the target delayed reading time by using the delayed database comprises: identifying the historical data which has been restored from the master database in the historical time period in the delayed database; copying the incremental data and the transaction log to the delayed database to cover the changed part of the historical data; searching for a target consistency point closest to the target delayed reading time from the archive log of the master database; restoring the incremental data in the delayed database; and playing back the transaction log to the target consistency point to obtain the restored data.

[0012] Another aspect of the present application provides a database data processing apparatus, comprising: a data identification module, configured to perform incremental change detection on a master database to identify incremental data that needs to be backed up; a time prediction module, configured to predict a backup start time of the incremental data based on historical transaction processing data of the master database and a data volume of the incremental data; and a data processing module, configured to perform backup on the incremental data according to the predicted backup start time, and restore the backed-up incremental data to a target delayed read time using a delayed database if the backup is completed.

[0013] Another aspect of the present application provides an electronic device, comprising: one or more processors; a memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described above.

[0014] Another aspect of the present application provides a computer-readable storage medium storing computer-executable instructions, which, when executed, implement the method as described above.

[0015] Another aspect of the present application provides a computer program product comprising computer-executable instructions, which, when executed, implement the method as described above.

[0016] According to embodiments of the present application, by performing incremental change detection on a master database, incremental data that needs to be backed up is accurately identified, thereby effectively reducing resource consumption caused by full database scanning. On this basis, in combination with historical transaction processing data and a data volume of current incremental data, a backup start time is predicted to select a period with low load of the master database to perform a backup task. In this way, the impact of backup on business performance can be maximally reduced, and the backup process can be ensured to be efficient and stable. After the backup is completed, time point recovery is implemented using the backed-up incremental data and a delayed database. In this way, the function of delayed read database of distributed database is implemented, which can ensure data consistency and maximize resource utilization. BRIEF DESCRIPTION OF DRAWINGS

[0017] The above and other objects, features and advantages of the present application will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0018] Figure 1 An exemplary system architecture to which the database data processing method and apparatus of the present application can be applied is shown.

[0019] Figure 2 A flowchart of a database data processing method according to an embodiment of the present application is shown.

[0020] Figure 3A system architecture diagram of a distributed database in a database data processing method according to an embodiment of the present application is shown.

[0021] Figure 4 A flow architecture diagram in a database data processing method according to an embodiment of the present application is shown.

[0022] Figure 5 A block diagram of a database data processing apparatus according to an embodiment of the present application is shown.

[0023] Figure 6 A block diagram of an electronic device adapted to implement a database data processing method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. It is to be understood, however, that these descriptions are merely exemplary of the application and are intended to provide an overall understanding of the application. Many modifications and variations will be apparent to those of ordinary skill in the art from this description, which is to be construed as illustrative only. Furthermore, where a term is provided in the description and / or the claims, it is intended to encompass the common meaning of the term, as well as any specific definition that can be given to the term in this description.

[0025] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used herein, the term "includes" and tautological equivalents thereof, means that the claimed features, steps, operations, and / or components are present, but does not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein including technical and scientific terms have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein are not intended to limit the present application, and should be interpreted as consistent with the context of the specification.

[0027] In the case where expressions such as "at least one of A, B, and C, etc." are used, it generally should be interpreted to include any of one, all of the same, or a combination thereof, unless otherwise specifically stated. In other words, the expression "at least one of A, B, and C, etc." should be interpreted to include one of A or B or C; a combination of at least one of A and at least one of B, a combination of at least one of A, at least one of B, and at least one of C; etc.

[0028] In the embodiments of the present application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, storage, etc. of the data involved (for example, including but not limited to user personal information) comply with the relevant legal regulations, are used for legal purposes, and do not violate public order and good customs. In particular, necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain user personal information security and network security.

[0029] In the embodiments of the present application, the authorization or consent of the user is obtained before the user personal information is acquired or collected.

[0030] Embodiments of the present application provide a database data processing method, device, equipment and storage medium. The method comprises: performing incremental change detection on a master database to identify incremental data that needs to be backed up; predicting a backup starting time of the incremental data based on the data volume of historical transaction processing data and the incremental data of the master database; and performing backup on the incremental data according to the predicted backup starting time, so that the incremental data is restored to a target delay reading time by using a delay database when the backup is completed.

[0031] Figure 1 An exemplary system architecture to which the database data processing method and device of the present application can be applied is shown. It should be noted that, Figure 1 The shown is only an example of a system architecture to which the embodiments of the present application can be applied, to help those skilled in the art understand the technical content of the present application, but does not mean that the embodiments of the present application cannot be used in other devices, systems, environments or scenarios.

[0032] As Figure 1 The system architecture 100 according to the embodiment can include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105, as shown. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 can include various connection types, such as wired and / or wireless communication links, etc.

[0033] The user can use the first terminal device 101, the second terminal device 102, the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as an example).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 can be a server providing various services, such as a background management server supporting a website browsed by a user using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing results (such as a webpage, information, or data generated or obtained according to a user request) to the terminal device.

[0036] It should be noted that the database data processing method provided by the embodiment of the present application can generally be executed by the server 105. Correspondingly, the database data processing apparatus provided by the embodiment of the present application can generally be arranged in the server 105. The database data processing method provided by the embodiment of the present application can also be executed by a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Correspondingly, the database data processing apparatus provided by the embodiment of the present application can also be arranged in a server or a server cluster different from the server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or the server 105. Alternatively, the database data processing method provided by the embodiment of the present application can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the database data processing apparatus provided by the embodiment of the present application can also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0037] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network, and the server in the above embodiment is only an example. According to the implementation needs, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network, and the server.

[0038] Figure 2 A flowchart of the database data processing method according to the embodiment of the present application is shown.

[0039] As Figure 2As shown, the method includes operations S210-S230.

[0040] In operation S210, incremental change detection is performed on the master database to identify incremental data that needs to be backed up.

[0041] In operation S220, a backup start time of the incremental data is predicted based on the historical transaction data of the master database and the data volume of the incremental data.

[0042] In operation S230, backup is performed on the incremental data according to the predicted backup start time, so that the backed up incremental data is restored to a target delayed reading time by the delayed database when the backup is completed.

[0043] According to an embodiment of the present application, in the management system of a distributed database, the data of the master database needs to be periodically backed up periodically. The core of this mechanism is that when sudden conditions such as data loss, fault recovery or business misoperation occur, the data can be quickly and reliably restored, thereby ensuring the continuity of the business.

[0044] When the first backup is performed, the full physical backup mechanism needs to be triggered to collect the data of each shard of the master database. After the first full backup is completed, the subsequent backup will be switched to the incremental backup mode. When performing incremental backup, incremental change detection is performed on the master database to accurately identify the incremental data that needs to be backed up. Incremental backup only captures the data that has changed since the last backup (whether full or incremental), including new, modified, deleted records, and corresponding transaction log increments. This way, the amount of backup data and transmission overhead is greatly reduced, ensuring data integrity while significantly improving backup efficiency.

[0045] After identifying the incremental data, its data volume needs to be determined. The data volume is calculated based on the size of the data page that has changed in the master database. For example, if the size of a single data page is 8KB and 10 data pages have changed, the data volume is 80KB. This way of calculating based on data pages can accurately quantify data changes and provide a reliable basis for subsequent backup strategies.

[0046] At the same time, the historical transaction data of the master database needs to be obtained, which contains the transaction processing volume. These historical data records the transaction processing scale and frequency of the database at different time periods, reflecting the change law of system load.

[0047] According to historical transaction processing data and data volume, a backup starting time of the incremental data is predicted. By analyzing peak and trough periods of the historical transaction processing data and combining the size of the data volume, a time point with relatively idle database resources can be selected to start the backup to avoid the system high load period. In this way, the influence of the backup task on normal business operation can be reduced, and the backup efficiency and system stability can be ensured. Subsequently, a backup task is initiated according to the predicted backup starting time to backup the incremental data.

[0048] Figure 3 A system architecture diagram of a distributed database in a database data processing method according to an embodiment of the application is shown.

[0049] As shown in Figure 3 In the system architecture of the distributed database, the uppermost layer of the architecture connects the management 301 and the application driver 302, which bear different responsibilities. The connection management 301 is responsible for processing connection-related operations of the client and the database system, ensuring the stability and reliability of the connection. The application driver 302 transmits requests and responses between the application program and the database system, playing the role of an intermediary.

[0050] The middle layer includes the coordinator 303 and the global transaction manager 304. The coordinator 303 is composed of a plurality of components, from 1 to m, m being a positive integer, and its main function is to receive requests from the connection management 301 and reasonably distribute these requests to the data nodes 305 for processing. Multiple coordinators 303 help to achieve load balancing and high availability. The global transaction manager 304 is composed of a master node and a slave node. The master node is responsible for managing transaction operations in the entire system, such as the initiation, submission and rollback of transactions, etc. The slave node acts as a backup and can take over the work of the master node in the event of a failure, ensuring the continuity of transaction management.

[0051] The lowermost layer is the data node 305, which is composed of a plurality of high-availability groups, such as high-availability group 1, high-availability group 2, …, high-availability group n, n being a positive integer. Each high-availability group adopts a master-slave structure, such as the master node and the slave node in high-availability group-1. The master node is responsible for processing read and write requests and synchronizing data changes to the slave node. The slave node is used for backup and recovery in the event of a master node failure. The design of such a high-availability group ensures that the system can automatically switch and maintain data availability in the event of a node failure.

[0052] The entire architecture fully embodies the key features of a distributed database system. Through the master-slave structure of the multiple coordinators, the global transaction manager 304, and the high-availability group design of the data nodes 305, high availability is achieved, ensuring that the system can still operate normally when some components fail. At the same time, the multiple coordinators share the request processing work, achieving load balancing and optimizing system performance. In addition, the global transaction manager 304 guarantees the correct execution of transactions and the consistency of data in a distributed environment, and the master-slave synchronization mechanism between the data nodes 305 ensures the consistency of data on different nodes.

[0053] As the amount of data stored in a distributed manner grows exponentially, the amount of business data also continues to rise. In order to reduce the pressure on the main database, a delay database can be set up to provide delay read functions, such as reading data from the previous day. This mechanism can meet the following needs: if business data is misoperated, the previous data can be obtained from the delay database for recovery. If the business needs to generate reports based on old data (such as generating yesterday's report today), the required data can also be extracted from the delay database.

[0054] Taking the data read time t1 and t2 provided by the delay database each day as an example, if t1 is 6 am and t2 is 6 pm, the data read by the user between 6 am and 6 pm is the data of the previous day t3 (such as 0 am). Among them, t3 is the target delay read time of the delay database. Therefore, the base backup of the main database needs to be completed before t3 time, so that the delay database can restore the backup data to the target delay read time. The recovery work needs to be completed before t1 time to ensure data readability. After confirming the completion of the incremental data backup, the delay database is used to restore the backup incremental data to the target delay read time, thereby ensuring the consistency and availability of the data.

[0055] According to the embodiments of the present application, by detecting the incremental changes of the main database, the incremental data that needs to be backed up is accurately identified, thereby effectively reducing the resource consumption caused by full library scanning. On this basis, combined with the data volume of historical transaction processing data and current incremental data, the starting time of backup is predicted to select a period with low load of the main database to perform the backup task. In this way, the impact of backup on business performance can be maximized, and the backup process can be ensured to be efficient and stable. After the backup is completed, the incremental data and the delay database are used to realize the point-in-time recovery. In this way, the function of the delay read library of the distributed database is realized, which can ensure the consistency of the data and maximize the utilization of resources.

[0056] According to an embodiment of the present application, the incremental change detection of the master database is performed to identify the incremental data that needs to be backed up, comprising: obtaining a change bitmap of the master database, wherein the master database records the block numbers of the data pages that are modified in a transaction in the change bitmap during a transaction commit phase; loading the corresponding data pages from the storage unit of the master database based on the block numbers recorded in the change bitmap to obtain the incremental data.

[0057] According to an embodiment of the present application, in a distributed database system, the master database pre-constructs and maintains a change bitmap data structure as a core tool for tracking data changes. The change bitmap is essentially a binary array, and each binary bit corresponds to a data page in the database. The state of "0" and "1" indicates whether the data page is changed.

[0058] When a transaction enters the commit phase, the database engine automatically scans the data operations involved in the transaction, locates all modified data pages, and records the corresponding data page block numbers in the change bitmap. For example, if the transaction modifies data pages numbered 10, 25, and 37, the 10th, 25th, and 37th bits in the change bitmap are marked as "1", and the bits corresponding to the other unchanged data pages remain as "0". This real-time marking mechanism ensures that the change bitmap can dynamically reflect the latest modification state of the database.

[0059] When performing incremental backup, first obtain the latest change bitmap, and extract all changed data page block numbers by scanning the bits with value "1" in the bitmap. Then, according to these block numbers, directly access the storage unit (such as disk partition, storage block device) of the master database, and selectively load the corresponding data pages. Since the change bitmap has accurately locked the data modification range, the system does not need to traverse the entire database, but only needs to read the data pages marked as changed, so that the incremental data can be quickly and completely obtained. This incremental data extraction method based on change bitmap greatly reduces the read and write operations and data transmission volume, significantly improves the backup efficiency, and reduces the impact on the database performance.

[0060] According to an embodiment of the present application, the database data processing method further comprises: detecting the number of block numbers recorded in the change bitmap; in a case where it is determined that the number of block numbers is less than a preset threshold, maintaining a polling waiting state until the number of recorded block numbers increases to the preset threshold; in a case where it is determined that the number is greater than or equal to the preset threshold, performing an operation of loading the corresponding data pages from the storage unit of the master database based on the plurality of block numbers recorded in the change bitmap.

[0061] According to an embodiment of the present application, in the incremental backup mechanism of the distributed database, the number of block numbers recorded in the change bitmap is monitored in real time to optimize the backup triggering strategy.

[0062] Specifically, the number of block numbers marked as "1" in the change bitmap is detected periodically through a timer or event listening mechanism. The preset threshold is an empirical value set in advance according to factors such as database performance, network bandwidth, and business load, for example, 1000 block numbers, and its core role is to avoid resource waste caused by frequent processing of a small amount of incremental data.

[0063] When the number of block numbers is detected to be less than the preset threshold, a polling waiting state is maintained, and the update of the change bitmap is continuously monitored. During this period, if a newly submitted transaction involves data page modification, the corresponding block number will be written into the change bitmap in real time, and the number of block numbers will gradually accumulate. The polling mechanism sets a reasonable time interval (such as once per second) to ensure real-time performance while avoiding excessive consumption of processor resources.

[0064] Once the number of block numbers is detected to reach or exceed the preset threshold, the polling is terminated immediately, and the incremental data loading process is triggered. At this time, based on all the block numbers recorded in the change bitmap, the database engine will batch read the corresponding physical data pages from the storage unit. This "threshold triggering" mechanism can combine multiple small-scale incremental changes into a batch operation, reduce the frequency of starting backup tasks, and reduce the fragmentation level of read-write operations, thereby improving the overall backup efficiency, while avoiding the impact of frequent backups on the submission performance of business transactions. By dynamically balancing the data change frequency and system resource utilization, this strategy can effectively optimize the backup energy efficiency of the distributed database.

[0065] According to an embodiment of the present application, based on the data volume of the historical transaction processing data and the incremental data of the master database, the backup starting time of the incremental data is predicted, including: determining the estimated backup duration according to the historical backup rate of the master database and the data volume of the incremental data; statistics of the transaction processing volume in the historical transaction processing data is performed to generate a transaction processing rate curve; the trend change characteristics of the transaction processing volume are extracted by analyzing the transaction processing rate curve; the backup starting time of the incremental data is predicted according to the trend change characteristics and the estimated backup duration, wherein the backup ending time determined based on the backup starting time and the estimated backup duration is earlier than the target delay reading time.

[0066] According to an embodiment of the present application, in the incremental backup strategy of the distributed database, the intelligent prediction of the backup time is realized by integrating historical data and real-time information. The historical backup records of the master database are called, the data transmission volume and time consumption in each backup process are analyzed, and the historical backup rate is calculated. Combined with the data volume of the current incremental data, the estimated backup duration is obtained through simple division operation, that is, "estimated backup duration = incremental data volume ÷ historical backup rate". This calculation process considers the key factors affecting the backup speed such as the read-write performance of the storage device and the network bandwidth, providing a quantitative basis for subsequent decision-making.

[0067] Meanwhile, the historical transaction processing data is deeply analyzed to count the transaction processing amount in different time periods in time dimension. Through data smoothing processing and visualization tools, a transaction processing rate curve (TPS curve, Transactions Per Second) is generated. The curve not only intuitively shows the fluctuation of the transaction processing amount in a day, a week or even a longer period, but also reveals the distribution law of business peaks and troughs, such as the high-load period in the daytime of weekdays and the low-load period at night.

[0068] Further, data mining and machine learning algorithms are used to extract features from the transaction processing rate curve. By identifying periodic fluctuations, abrupt points, rising or falling trends and other features in the curve, a change model of the transaction processing amount is constructed. For example, the sliding window algorithm is used to detect the change slope of the recent transaction processing amount to determine whether the system load is in an ascending or descending period, so as to predict the load trend in the future period.

[0069] Based on the above analysis results, the optimal backup starting time is reversely calculated based on the core constraint condition that the backup end time is earlier than the target delayed reading time, combined with the estimated backup duration. Specifically, the low period in the transaction processing rate curve is preferentially selected to minimize the impact of the backup operation on the business. At the same time, a sufficient time window is reserved to ensure that the backup task is completed before the target delayed reading time. For example, if the estimated backup duration is 30 minutes and the target delayed reading time is 2 a.m., a time point with low transaction processing amount and no high load within 30 minutes from the time point is found, such as 1:15 a.m., as the starting time of the incremental backup. Through this data-driven dynamic decision-making mechanism, the backup task and business operation can be efficiently coordinated while ensuring data timeliness.

[0070] According to the embodiments of the present application, the backup starting time of the incremental data is predicted according to the trend change feature and the estimated backup duration, including: based on the trend change feature, a plurality of candidate time intervals with transaction processing amount lower than a preset number are identified from the time processing sequence of the transaction processing curve; from the plurality of candidate time intervals, a target time interval earlier than the target delayed reading time and closest to the target delayed reading time is determined; and the backup starting time is reversely calculated based on the end boundary time of the target time interval and combined with the estimated backup duration.

[0071] According to the embodiments of the present application, the backup starting time is determined based on the transaction processing trend feature through multi-dimensional screening and accurate calculation. First, the transaction processing rate curve is analyzed in detail, and a reasonable transaction processing amount preset number is set according to the historical data fluctuation range combined with statistical methods and dynamic threshold algorithms.

[0072] For example, by calculating the average and standard deviation of the transaction volume in the past week, the preset number is set to 30% of the average value to accommodate the natural fluctuations in business load. Subsequently, the sliding window technique is used to traverse the time processing sequence of the transaction curve, and the transaction volume in each time slice is monitored in real time. When the transaction volume in a certain time interval is continuously lower than the preset number, the interval is marked as a candidate time interval, and information such as the start and end time and the average transaction volume is stored in the candidate list.

[0073] After obtaining multiple candidate time intervals, the candidate list is sorted according to the dual screening conditions of "earlier than the target delay reading time and closest". By calculating the time difference between the end boundary time of each candidate time interval and the target delay reading time, and eliminating the intervals that exceed the target time, the interval with the smallest time difference is selected as the target time interval.

[0074] For example, if the target delay reading time is 3 a.m., there are two candidate intervals - [1 a.m. - 1:30 a.m.] and [2 a.m. - 2:30 a.m.], the latter is selected as the target time interval, ensuring that the backup task can avoid business peak and complete as close to the target time as possible.

[0075] After determining the target time interval, the end boundary time of the target time interval is used as the reference, and the estimated backup duration calculated in the previous step is used for reverse calculation. For example, if the target time interval is [2 a.m. - 2:30 a.m.] and the estimated backup duration is 20 minutes, the system sets the backup start time to 2:10 a.m., ensuring that the backup task starts within the target time interval and ends before the end boundary time, effectively reducing the impact of the backup operation on the business system, while meeting the timeliness requirements of data delay reading.

[0076] According to the embodiment of the present application, the database data processing method further comprises: in the case of backup completion, creating a data consistency point based on the backup end time; and storing the data consistency point and the transaction log corresponding to the incremental data generated by the primary database in association.

[0077] According to the embodiment of the present application, after the distributed database system completes the incremental backup, the data consistency marking and associated storage operation is automatically executed to ensure the integrity and traceability of the backup data. At the moment when the backup task is completed, the backup end time is accurately recorded based on the current system clock, and this time is used as a key data consistency point (Consistency Point). This timestamp not only identifies the time boundary of the backup data, but also serves as a reference point for subsequent data recovery and verification. The creation of the data consistency point uses atomic operation to ensure the accuracy of the record is not disturbed by concurrent transactions.

[0078] Subsequently, the transaction logs related to the incremental data are automatically retrieved and associated. In a distributed database architecture, each transaction generates detailed log records during execution, including operation type (add, delete, modify), data change content, execution timestamp, and other information. The system filters all transaction logs involved in the generation of the backup data from the log storage system by analyzing the transaction identifiers or timestamp ranges contained in the incremental data. These logs record all data change operations from the last backup to the current consistency point and are the key basis for implementing point-in-time recovery.

[0079] Finally, the data consistency point is associated with the corresponding transaction log for storage. This association is usually saved in the form of metadata, such as creating an index file in a distributed file system or establishing a mapping table in a relational database. The structure design of associated storage needs to meet the needs of efficient query and fast recovery, such as indexing transaction logs by timestamp or establishing a many-to-many association between consistency points and transaction log batches.

[0080] In this way, when data recovery is needed, the system can quickly locate the corresponding consistency point according to the specified time point, then load the associated transaction log, gradually replay the data change process, and accurately restore the database state to the target time point, thereby ensuring the data reliability and business continuity of the distributed database in the face of failures or data damage.

[0081] According to an embodiment of the present application, the incremental data is restored to the target delay reading time by using a delay database, including: identifying historical data in a historical time period that has been restored from a master database in the delay database; copying the incremental data and the transaction log to the delay database to cover the changed part of the historical data; finding the target consistency point closest to the target delay reading time from the archive log of the master database; restoring the incremental data in the delay database; and replaying the transaction log to the target consistency point to obtain the restored data.

[0082] According to an embodiment of the present application, in the delay recovery mechanism of a distributed database, the time point synchronization between the delay database and the master database is achieved through accurate data comparison and incremental update.

[0083] First, the delay database is scanned for metadata to identify historical data that has been restored from the master database in a historical time period. This process locates all data blocks that have completed the restoration operation and their corresponding time range by querying the restoration record table or timestamp index of the delay database. For example, by analyzing the historical restoration record table in the delay database, the start time, end time, and affected data page range of each restoration operation are obtained to construct a complete view of historical data.

[0084] Based on the above identification result, the incremental data of this backup and the associated transaction log are copied to the delayed database. During the data copying process, a block-level comparison algorithm (such as hash check) is used to accurately identify the changed part of the historical data, and only these parts are updated. This embodiment identifies by reading the modification latest log sequence number of the data page, can efficiently find the change data of the last day, and greatly reduces the data moving amount. For example, the system calculates the hash value of the data page, compares the hash values of the corresponding data pages in the main database and the delayed database, and only replaces the inconsistent data pages, thereby significantly reducing the data transmission amount and storage overhead.

[0085] At the same time, the target consistency point closest to the target delayed reading time from the archive log of the main database is retrieved. The archive log is a complete record of database transactions, including the exact timestamp of each transaction submission. Through the binary search algorithm, the transaction record with the closest timestamp to the target delayed reading time is quickly located, and it is marked as the target consistency point. For example, if the target delayed reading time is 2 a.m., the transaction identifier with the closest timestamp but not later than 2 a.m. in the archive log is found as the target consistency point.

[0086] After completing the data copying and target consistency point positioning, the recovery operation is performed in the delayed database. First, the incremental data is loaded and written to the corresponding storage location of the delayed database. Then, according to the time sequence of the transaction log, the transaction is played back from the starting point corresponding to the incremental data, and the change operation of each transaction is gradually applied until the target consistency point is reached. During this process, the atomicity and isolation principles of transactions are strictly followed to ensure the accuracy of data recovery. Finally, the data state in the delayed database will be accurately restored to the target delayed reading time, providing a reliable data foundation for subsequent delayed read operations.

[0087] Figure 4 The flow architecture diagram in the database data processing method according to an embodiment of the application is shown.

[0088] As Figure 4 shown, the database backup and recovery process architecture adopts modular design, and each component works together to ensure data security. The log archiving server 401 as the core component specially stores the operation logs of the main database 404, which record all data change history and provide key basis for subsequent backup and recovery.

[0089] The backup estimation module 402 performs intelligent analysis based on the data of the main database 404, dynamically determines the optimal backup strategy and resource requirements by evaluating indicators such as log data volume and operation frequency, and returns the output results to the main database 404. At the same time, the output of the backup estimation module 402 can guide the specific operation of the backup execution module 403 to ensure that the backup process is efficient and economical. It may analyze the data volume, operation frequency and other information in the log to determine the appropriate backup strategy and required resources. The backup execution module 403 extracts data from the main database 404 to perform backup tasks.

[0090] The recovery execution module 405 is responsible for accurately restoring backup data to the delay database 406 when needed. The delay database 406 is specially used to store historical data to support business scenarios with low real-time requirements such as data analysis.

[0091] The entire architecture forms a complete closed loop from log management to backup estimation, from backup execution to final recovery, which not only guarantees data security, but also meets the diverse business needs. Through this design, the system can effectively prevent data loss risks and provide flexible data support for different business scenarios.

[0092] Figure 5 A block diagram of a database data processing apparatus according to an embodiment of the application is shown.

[0093] As shown in Figure 5 The database data processing apparatus 500 includes a data recognition module 510, a time prediction module 520, and a data processing module 530.

[0094] The data recognition module 510 is configured to perform incremental change detection on the main database to identify incremental data that needs to be backed up.

[0095] The time prediction module 520 is configured to predict a backup start time of the incremental data based on historical transaction processing data and a data volume of the incremental data of the main database.

[0096] The data processing module 530 is configured to perform backup on the incremental data according to the predicted backup start time, and to restore the backed up incremental data to a target delay reading time using the delay database in the case where the backup is completed.

[0097] According to an embodiment of the application, the data recognition module 510 includes a change acquisition submodule and a data loading submodule.

[0098] The change acquisition submodule is configured to acquire a change bitmap of the main database, wherein the main database records a block number of a data page that is modified by a transaction to the change bitmap at a transaction commit stage.

[0099] The data loading submodule is configured to load corresponding data pages from the storage unit of the primary database based on the plurality of block numbers recorded in the change bitmap, to obtain the incremental data.

[0100] According to an embodiment of the present application, the database data processing apparatus 500 further comprises a quantity detection module, a polling waiting module and an operation execution module.

[0101] The quantity detection module is configured to detect the quantity of block numbers recorded in the change bitmap.

[0102] The polling waiting module is configured to maintain a polling waiting state until the quantity of block numbers recorded increases to the preset threshold, if it is determined that the quantity of block numbers is less than the preset threshold.

[0103] The operation execution module is configured to execute an operation of loading corresponding data pages from the storage unit of the primary database based on the plurality of block numbers recorded in the change bitmap, if it is determined that the quantity is greater than or equal to the preset threshold.

[0104] According to an embodiment of the present application, the time prediction module 520 comprises a duration determination submodule, a curve generation submodule, a trend analysis submodule and a time prediction submodule.

[0105] The duration determination submodule is configured to determine an estimated backup duration according to a historical backup rate of the primary database and a data volume of the incremental data.

[0106] The curve generation submodule is configured to count transaction processing volumes in the historical transaction processing data, to generate a transaction processing rate curve.

[0107] The trend analysis submodule is configured to analyze the transaction processing rate curve, to extract a trend change feature of the transaction processing volumes.

[0108] The time prediction submodule is configured to predict a backup starting time of the incremental data according to the trend change feature and the estimated backup duration, wherein a backup ending time determined based on the backup starting time and the estimated backup duration is earlier than the target delayed reading time.

[0109] According to an embodiment of the present application, the time prediction submodule comprises a candidate identification unit, a target determination unit and a time determination unit.

[0110] The candidate identification unit is configured to identify a plurality of candidate time intervals with transaction processing volumes lower than a preset quantity from a time processing sequence of the transaction processing curve, based on the trend change feature.

[0111] The target determination unit is configured to determine a target time interval from the plurality of candidate time intervals, which is earlier than the target delayed reading time and closest to the target delayed reading time.

[0112] The time determination unit is configured to determine a backup starting time based on a target time interval end boundary time and a backup time length.

[0113] According to an embodiment of the present application, the database data processing apparatus 500 further comprises a consistency creation module and a log storage module.

[0114] The consistency creation module is configured to create a data consistency point based on a backup end time when the backup is completed.

[0115] The log storage module is configured to store the data consistency point and a transaction log corresponding to the incremental data generated by the primary database in association.

[0116] According to an embodiment of the present application, the data processing module 530 comprises a history identification sub-module, a data copy sub-module, a consistency search sub-module and a data recovery sub-module.

[0117] The history identification sub-module is configured to identify historical data that has been recovered from the primary database in a historical time period in the delay database.

[0118] The data copy sub-module is configured to copy the incremental data and the transaction log to the delay database to overwrite the changed part of the historical data.

[0119] The consistency search sub-module is configured to search for a target consistency point closest to a target delay reading time from the archive log of the primary database.

[0120] The data recovery sub-module is configured to recover the incremental data in the delay database, and play back the transaction log to the target consistency point to obtain the recovery data.

[0121] Any one or more of the modules, sub-modules, units, sub-units, or at least part of the functions of any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be implemented in one module. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be split into multiple modules for implementation. Any one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware through integration or packaging of circuits, or in any one of software, hardware and firmware or in a proper combination of any one or more of them. Alternatively, one or more of the modules, sub-modules, units, sub-units according to the embodiments of the present application can be at least partially implemented as computer program modules that can perform corresponding functions when executed.

[0122] For example, any of the data identifying module 510, the time predicting module 520 and the data processing module 530 can be combined in one module / unit / sub-unit, or any of the modules / units / sub-units can be split into multiple modules / units / sub-units. Or, at least part of the function of one or more of the modules / units / sub-units can be combined with at least part of the function of other modules / units / sub-units, and implemented in one module / unit / sub-unit. According to an embodiment of the present application, at least one of the data identifying module 510, the time predicting module 520 and the data processing module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. in hardware or firmware, or implemented in any one of software, hardware and firmware or in a proper combination of any of them. Or, at least one of the data identifying module 510, the time predicting module 520 and the data processing module 530 can be at least partially implemented as a computer program module which, when executed, can perform the corresponding function.

[0123] It should be noted that the database data processing apparatus part in the embodiments of the present application corresponds to the database data processing method part in the embodiments of the present application, and the description of the database data processing apparatus part is specifically referred to the database data processing method part, which will not be repeated here.

[0124] Figure 6 A block diagram of an electronic device suitable for implementing the database data processing method according to an embodiment of the present application is shown. Figure 6 The electronic device shown is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0125] As shown in Figure 6 The electronic device 600 according to an embodiment of the present application includes a processor 601 which can perform various appropriate actions and processes according to programs stored in a read only memory (ROM) 602 or loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 can include, for example, a general purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special purpose microprocessor (such as an application specific integrated circuit (ASIC)), etc. The processor 601 can also include an on-board memory for cache use. The processor 601 can include a single processing unit or multiple processing units for performing different actions of the method processes according to embodiments of the present application.

[0126] In the RAM 603, various programs and data required for the operation of the electronic device 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via the bus 604. The processor 601 performs various operations of the method flow according to the embodiments of the present application by executing the programs in the ROM 602 and / or the RAM 603. Note that the programs can also be stored in one or more memories other than the ROM 602 and the RAM 603. The processor 601 can also perform various operations of the method flow according to the embodiments of the present application by executing the programs stored in the one or more memories.

[0127] According to the embodiments of the present application, the electronic device 600 can further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 can further include one or more of the following components connected to the input / output (I / O) interface 605: an input part 606 including a keyboard, a mouse, and the like; an output part 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage part 608 including a hard disk, and the like; and a communication part 609 including a network interface card such as a LAN card, a modem, and the like. The communication part 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as necessary. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 610 as necessary, so that a computer program read therefrom is installed in the storage part 608 as necessary.

[0128] According to the embodiments of the present application, the method flow according to the embodiments of the present application can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-described functions defined in the system of the embodiments of the present application are performed. According to the embodiments of the present application, the system, the device, the apparatus, the module, the unit, and the like described above can be implemented by computer program modules.

[0129] The application further provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist independently without being assembled into the device / apparatus / system. The computer readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the application.

[0130] According to the embodiments of the application, the computer readable storage medium can be a non-volatile computer readable storage medium. For example, it can include but is not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in connection with an instruction execution system, apparatus, or device.

[0131] For example, according to the embodiments of the application, the computer readable storage medium can include one or more memories of the ROM 602 and / or the RAM 603 described above and / or other than the ROM 602 and the RAM 603.

[0132] The embodiments of the application also include a computer program product, which includes a computer program containing program codes for executing the method provided by the embodiments of the application, and when the computer program product is run on an electronic device, the program codes are used to make the electronic device implement the database data processing method provided by the embodiments of the application.

[0133] When the computer program is executed by the processor 601, the above functions defined in the system / apparatus of the embodiments of the application are executed. According to the embodiments of the application, the system, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0134] In one embodiment, the computer program can rely on tangible storage media such as optical storage media, magnetic storage media, etc. In another embodiment, the computer program can also be transmitted, distributed, downloaded and installed in the form of signals on a network medium, and be downloaded and installed through the communication part 609 and / or installed from the detachable medium 611. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any suitable combination of the foregoing.

[0135] According to embodiments of the present application, program code for implementing the computer programs provided by embodiments of the present application can be written in any combination of one or more programming languages, and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The programming language includes, but is not limited to, such languages as Java, C++, python, "C" language, or the like. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0136] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0137] The embodiments of the present application have been described above. However, these embodiments are merely for the purpose of illustration and are not intended to limit the scope of the present application. Although the respective embodiments are described above separately, this does not mean that the measures in the respective embodiments cannot be used advantageously in combination. Various alternatives and modifications to disclosed embodiments can be apparent to those skilled in the art in view of the foregoing without departing from the scope of the present application. Such alternatives and modifications are intended to fall within the scope of the application.

Claims

1. A database data processing method characterized by, The method comprises: Incremental change detection is performed on the master database to identify incremental data that needs to be backed up; Based on the historical transaction data of the master database and the data volume of the incremental data, a backup start time of the incremental data is predicted; According to the predicted backup start time, backup is performed on the incremental data, and if the backup is completed, the backed-up incremental data is restored to a target delayed reading time by using a delayed database; The prediction of the backup start time of the incremental data based on the historical transaction data of the master database and the data volume of the incremental data comprises: According to the historical backup rate of the master database and the data volume of the incremental data, an estimated backup duration is determined; The transaction volume in the historical transaction data is counted to generate a transaction rate curve; The trend change characteristics of the transaction volume are extracted by analyzing the transaction rate curve; According to the candidate time interval in which the transaction volume is lower than a preset number in the trend change characteristics and the estimated backup duration, the backup start time of the incremental data is predicted, wherein the backup end time determined based on the backup start time and the estimated backup duration is earlier than the target delayed reading time.

2. The method of claim 1, wherein, The incremental change detection performed on the master database to identify the incremental data that needs to be backed up comprises: A change bitmap of the master database is obtained, wherein the master database records the block number of the data page that is modified in a transaction in the change bitmap during a transaction submission stage; Based on the plurality of block numbers recorded in the change bitmap, the corresponding data pages are loaded from the storage unit of the master database to obtain the incremental data.

3. The method of claim 2, wherein, The method further comprises: The number of block numbers recorded in the change bitmap is detected; If it is determined that the number of block numbers is less than a preset threshold, a polling waiting state is maintained until the number of recorded block numbers increases to the preset threshold; If it is determined that the number of block numbers is greater than or equal to the preset threshold, the operation of loading the corresponding data pages from the storage unit of the master database based on the plurality of block numbers recorded in the change bitmap is performed.

4. The method of claim 1, wherein, The prediction of the backup start time of the incremental data according to the candidate time interval in which the transaction volume is lower than a preset number in the trend change characteristics and the estimated backup duration comprises: Based on the trend change characteristics, a plurality of candidate time intervals in which the transaction volume is lower than a preset number are identified from the time processing sequence of the transaction curve; From the plurality of candidate time intervals, a target time interval in which the end boundary time of the candidate time interval is earlier than the target delayed reading time and is closest to the target delayed reading time is determined; The backup start time is reversely calculated based on the end boundary time of the target time interval and in combination with the estimated backup duration.

5. The method of claim 1, wherein, The method further comprises: If the backup is completed, a data consistency point is created based on the backup end time; The data consistency point and the transaction log corresponding to the incremental data generated by the master database are stored in association.

6. The method of claim 5, wherein, The restoring the backed up incremental data to a target delayed reading time point by using the delayed database comprises: identifying historical data in a historical time period that has been restored from the master database in the delayed database; copying the incremental data and the transaction log to the delayed database to overwrite the changed part of the historical data; finding a target consistency point closest to the target delayed reading time point from the archive log of the master database; restoring the incremental data in the delayed database and replaying the transaction log to the target consistency point to obtain the restored data.

7. A database data processing apparatus, characterized by comprising: The apparatus comprises: a data identification module configured to perform incremental change detection on a master database to identify incremental data that needs to be backed up; a time point prediction module configured to predict a backup start time point of the incremental data based on historical transaction processing data of the master database and a data volume of the incremental data; a data processing module configured to perform backup on the incremental data according to the predicted backup start time point, and to restore the backed up incremental data to a target delayed reading time point by using a delayed database in the case that the backup is completed; a time length determination submodule configured to determine an estimated backup time length according to a historical backup rate of the master database and the data volume of the incremental data; a curve generation submodule configured to generate a transaction processing rate curve by counting transaction processing volumes in the historical transaction processing data; a trend analysis submodule configured to extract trend change characteristics of the transaction processing volumes by analyzing the transaction processing rate curve; a time point prediction submodule configured to predict the backup start time point of the incremental data according to a candidate time interval in which the transaction processing volumes are lower than a preset number in the trend change characteristics and the estimated backup time length, wherein a backup end time point determined based on the backup start time point and the estimated backup time length is earlier than the target delayed reading time point.

8. An electronic device, comprising: comprise: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 6.

Images