Database data processing method and device, equipment and storage medium

Abstract

The invention provides a database data processing method and device, equipment and a storage medium. The method can be applied to the cloud computing field. The method comprises the following steps: performing incremental change detection on a main database to identify incremental data needing to be backed up; predicting a backup starting moment of the incremental data based on the historical transaction processing data of the main database and the data volume of the incremental data; and performing backup on the incremental data according to the predicted backup starting moment so as to recover the backed-up incremental data to the target delay reading moment by utilizing the delay database under the condition that the backup is completed.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a database data processing method, apparatus, device and storage medium. Background Art

[0002] In a distributed database management system, the master database data needs to be backed up regularly. As the amount of data in distributed storage continues to grow exponentially, a delayed database architecture is often used to synchronize master database data in real time and provide delayed read capabilities.

[0003] In the process of realizing the concept of the present invention, there are at least the following problems in the related art: when processing large-scale data, delaying the real-time synchronization of the database will consume a large amount of system resources, resulting in a long synchronization process, which not only affects the operating efficiency of the main database, but also reduces the speed of data recovery, and ultimately leads to a decrease in overall resource utilization. Summary of the Invention

[0004] In view of this, the present invention provides a database data processing method, apparatus, device and storage medium.

[0005] One aspect of the present invention provides a database data processing method, comprising: performing incremental change detection on a primary database to identify incremental data that needs to be backed up; predicting a backup start time for the incremental data based on historical transaction processing data of the primary database and the amount of the incremental data; performing a backup of the incremental data according to the predicted backup start time, so that when the backup is completed, the backed-up incremental data is restored to a target delayed reading time using a delayed database.

[0006] According to an embodiment of the present invention, the above-mentioned incremental change detection on the main database to identify the incremental data that needs to be backed up includes: obtaining the change bitmap of the above-mentioned main database, wherein, during the transaction commit phase, the above-mentioned main database will record the block numbers of the data pages where the transaction modifications occur in the above-mentioned change bitmap; based on the multiple block numbers recorded in the above-mentioned change bitmap, the corresponding data pages are loaded from the storage unit of the above-mentioned main database to obtain the above-mentioned incremental data.

[0007] According to an embodiment of the present invention, the above method also includes: detecting the number of block numbers recorded in the above change bitmap; when it is determined that the number of block numbers is less than a preset threshold, maintaining a polling waiting state until the number of recorded block numbers increases to the above preset threshold; when it is determined that the above number is greater than or equal to the above preset threshold, executing the above operation of loading the corresponding data pages from the storage unit of the above main database based on the multiple block numbers recorded in the above change bitmap.

[0008] According to an embodiment of the present invention, the above-mentioned prediction of the backup start time of the above-mentioned incremental data based on the historical transaction processing data of the above-mentioned main database and the data volume of the above-mentioned incremental data includes: determining the estimated backup duration according to the historical backup rate of the above-mentioned main database and the data volume of the above-mentioned incremental data; performing statistics on the transaction processing volume in the above-mentioned historical transaction processing data to generate a transaction processing rate curve; extracting the trend change characteristics of the above-mentioned transaction processing volume by analyzing the above-mentioned transaction processing rate curve; predicting the backup start time of the above-mentioned incremental data according to the above-mentioned trend change characteristics and the above-mentioned estimated backup duration, wherein the backup end time determined based on the above-mentioned backup start time and the above-mentioned estimated backup duration is earlier than the above-mentioned target delayed reading time.

[0009] According to an embodiment of the present invention, the above-mentioned prediction of the backup start time of the above-mentioned incremental data based on the above-mentioned trend change characteristics and the above-mentioned estimated backup duration includes: based on the above-mentioned trend change characteristics, identifying a plurality of candidate time intervals in which the above-mentioned transaction processing volume is lower than a preset number from the time processing sequence of the above-mentioned transaction processing curve; determining a target time interval that is earlier than the above-mentioned target delayed reading time and closest to the above-mentioned target delayed reading time from the multiple above-mentioned candidate time intervals; taking the end boundary time of the above-mentioned target time interval as a reference, combined with the above-mentioned estimated backup duration, reversely calculating the above-mentioned backup start time.

[0010] According to an embodiment of the present invention, the above method also includes: when the backup is completed, creating a data consistency point based on the above backup end time; and associating and storing the above data consistency point and the transaction log corresponding to the above incremental data generated by the above primary database.

[0011] According to an embodiment of the present invention, the above-mentioned use of the delayed database to restore the backed-up incremental data to the target delayed reading time includes: identifying the historical data in the above-mentioned delayed database that has been restored from the above-mentioned main database within the historical time period; copying the above-mentioned incremental data and the above-mentioned transaction log to the above-mentioned delayed database to cover the changed part of the above-mentioned historical data; searching for the target consistency point closest to the above-mentioned target delayed reading time from the archive log of the above-mentioned main database; restoring the above-mentioned incremental data in the above-mentioned delayed database; and replaying the above-mentioned transaction log to the above-mentioned target consistency point to obtain restored data.

[0012] Another aspect of the present invention provides a database data processing device, including: a data identification module for performing incremental change detection on a main database to identify incremental data that needs to be backed up; a time prediction module for predicting the backup start time of the incremental data based on the historical transaction processing data of the main database and the data volume of the incremental data; a data processing module for performing a backup of the incremental data according to the predicted backup start time, so that when the backup is completed, the backed up incremental data can be restored to the target delayed reading time using a delayed database.

[0013] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0014] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, wherein the instructions are used to implement the above method when executed.

[0015] Another aspect of the present invention provides a computer program product, which includes computer executable instructions. When the instructions are executed, they are used to implement the method described above.

[0016] According to an embodiment of the present invention, by performing incremental change detection on the master database, the incremental data that needs to be backed up is accurately identified, thereby effectively reducing the resource consumption caused by full database scanning. On this basis, the starting time of the backup is predicted in combination with the historical transaction processing data and the data volume of the current incremental data, so as to select a period of time when the load on the master database is low to perform the backup task. This can minimize the impact of the backup on business performance and ensure that the backup process is both efficient and stable. After the backup is completed, the backed-up incremental data and the delayed database are used to achieve point-in-time recovery. Using this method to implement the function of delayed reading the distributed database can not only ensure data consistency but also maximize resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0018] Figure 1 An exemplary system architecture is shown to which the database data processing method and apparatus of the present invention can be applied.

[0019] Figure 2 A flowchart of a database data processing method according to an embodiment of the present invention is shown.

[0020] Figure 3A system architecture diagram of a distributed database in a database data processing method according to an embodiment of the present invention is shown.

[0021] Figure 4 A schematic diagram of the process architecture of a database data processing method according to an embodiment of the present invention is shown.

[0022] Figure 5 A block diagram of a database data processing device according to an embodiment of the present invention is shown.

[0023] Figure 6 A block diagram of an electronic device suitable for implementing a database data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0029] In the embodiment of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] Embodiments of the present invention provide a database data processing method, apparatus, device, and storage medium. The method includes: performing incremental change detection on a primary database to identify incremental data that needs to be backed up; predicting a backup start time for the incremental data based on historical transaction data and the amount of incremental data in the primary database; backing up the incremental data based on the predicted backup start time, and restoring the backed-up incremental data to a target delayed read time using a delayed database after the backup is complete.

[0031] Figure 1 The following shows an exemplary system architecture to which the database data processing method and apparatus of the present invention can be applied. Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present invention may be applied, to help those skilled in the art understand the technical content of the present invention, but do not mean that the embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0034] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0036] It should be noted that the database data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the database data processing device provided in the embodiment of the present invention can generally be set in the server 105. The database data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the database data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Alternatively, the database data processing method provided in the embodiment of the present invention can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the database data processing device provided in the embodiment of the present invention can also be set in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0037] It should be understood that Figure 1 The number of the first terminal device, the second terminal device, the third terminal device, the network and the server is only . According to the implementation requirements, there can be any number of the first terminal device, the second terminal device, the third terminal device, the network and the server.

[0038] Figure 2 A flowchart of a database data processing method according to an embodiment of the present invention is shown.

[0039] like Figure 2As shown, the method includes operations S210 to S230.

[0040] In operation S210 , incremental change detection is performed on the primary database to identify incremental data that needs to be backed up.

[0041] In operation S220 , a backup start time of the incremental data is predicted based on the historical transaction data of the primary database and the data volume of the incremental data.

[0042] In operation S230 , the incremental data is backed up according to the predicted backup start time, so that when the backup is completed, the backed up incremental data is restored to the target delayed reading time using the delayed database.

[0043] According to an embodiment of the present invention, in a distributed database management system, the master database data is backed up regularly. The core of this mechanism is to quickly and reliably restore data in the event of emergencies such as data loss, fault recovery, or business misoperation, thereby ensuring business continuity.

[0044] During the first backup, a full physical backup mechanism is triggered to collect all data from all shards of the master database. After the initial full backup, subsequent backups switch to incremental backup mode. During incremental backups, incremental change detection is performed on the master database to accurately identify the incremental data that needs to be backed up. Incremental backups only capture data that has changed since the last backup (whether full or incremental), including new, modified, and deleted records, as well as the corresponding transaction log increments. This approach significantly reduces the amount of backup data and transmission overhead, significantly improving backup efficiency while ensuring data integrity.

[0045] After identifying incremental data, its volume must be determined. This volume is calculated based on the size of the data pages that have changed in the primary database. For example, if a single data page is 8KB and 10 pages have changed, the data volume is 80KB. This data page-based calculation method accurately quantifies data changes and provides a reliable basis for subsequent backup strategies.

[0046] At the same time, it is necessary to obtain historical transaction data from the master database, including transaction volume. This historical data records the scale and frequency of database transactions over different time periods, reflecting the changing patterns of system load.

[0047] Based on historical transaction data and data volume, the backup start time for incremental data is predicted. By analyzing the peak and trough periods of historical transaction data and taking into account the data volume, backups can be initiated during periods of relatively low database resources, avoiding periods of high system load. This reduces the impact of backup tasks on normal business operations, ensuring backup efficiency and system stability. Subsequently, the backup task is initiated at the predicted backup start time to back up the incremental data.

[0048] Figure 3 A system architecture diagram of a distributed database in a database data processing method according to an embodiment of the present invention is shown.

[0049] like Figure 3 As shown in the figure, in the distributed database system architecture, the top-level connection management layer 301 and application driver 302 each have distinct responsibilities. Connection management layer 301 handles operations related to the client connection to the database system, ensuring the stability and reliability of the connection. Application driver 302 acts as an intermediary, transferring requests and responses between the application and the database system.

[0050] The middle layer includes the coordinator 303 and the global transaction manager 304. The coordinator 303 consists of multiple components, numbered from 1 to m, where m is a positive integer. Its primary function is to receive requests from the connection manager 301 and rationally distribute them to the data nodes 305 for processing. Multiple coordinators 303 help achieve load balancing and high availability. The global transaction manager 304 consists of a master node and slave nodes. The master node is responsible for managing transaction operations across the entire system, such as transaction initiation, commit, and rollback. The slave nodes serve as backups, taking over the master node's work if it fails, ensuring the continuity of transaction management.

[0051] At the bottom layer are data nodes 305, which are composed of multiple high-availability groups, such as high-availability group 1, high-availability group 2, ..., high-availability group n, where n is a positive integer. Each high-availability group uses a master-slave architecture. For example, high-availability group 1 contains a master node and slave nodes. The master node handles read and write requests and synchronizes data changes to the slave nodes, while the slave nodes are used for backup and recovery in the event of a master node failure. This high-availability group design ensures that the system can automatically failover in the event of a node failure, maintaining data availability.

[0052] The entire architecture fully embodies the key characteristics of a distributed database system. Through the master-slave structure of multiple coordinators, the global transaction manager 304, and the high-availability group design of data nodes 305, high availability is achieved, ensuring that the system can continue to operate normally even if some components fail. Furthermore, multiple coordinators share the workload of request processing, achieving load balancing and optimizing system performance. Furthermore, the global transaction manager 304 ensures the correct execution of transactions and data consistency in a distributed environment, while the master-slave synchronization mechanism between data nodes 305 ensures data consistency across different nodes.

[0053] As the amount of data in distributed storage grows exponentially, the volume of business data continues to climb. To alleviate pressure on the primary database, a delayed database can be set up to provide delayed read functionality, such as reading data from the previous day. This mechanism meets the following requirements: If business data is mishandled, the previous data can be retrieved from the delayed database for recovery. If the business needs to generate reports based on old data (for example, generating yesterday's report today), the required data can also be retrieved from the delayed database.

[0054] For example, if the delayed database provides daily data read times t1 and t2, and t1 is 6:00 AM and t2 is 6:00 PM, then data read between 6:00 AM and 6:00 PM is data from t3 of the previous day (e.g., midnight). t3 is the delayed database's target delayed read time. Therefore, the primary database's base backup must be completed before t3 so that the delayed database can restore the backed-up data to the target delayed read time. The restore must be completed before t1 to ensure data readability. After confirming the completion of the incremental data backup, the delayed database is used to restore the backed-up incremental data to the target delayed read time, ensuring data consistency and availability.

[0055] According to an embodiment of the present invention, by performing incremental change detection on the master database, the incremental data that needs to be backed up is accurately identified, thereby effectively reducing the resource consumption caused by full database scanning. On this basis, the starting time of the backup is predicted in combination with the historical transaction processing data and the data volume of the current incremental data, so as to select a period of time when the load on the master database is low to perform the backup task. This can minimize the impact of the backup on business performance and ensure that the backup process is both efficient and stable. After the backup is completed, the backed-up incremental data and the delayed database are used to achieve point-in-time recovery. Using this method to implement the function of delayed reading the distributed database can not only ensure data consistency but also maximize resource utilization.

[0056] According to an embodiment of the present invention, incremental change detection is performed on the primary database to identify incremental data that needs to be backed up, including: obtaining a change bitmap of the primary database, wherein, during the transaction commit phase, the primary database records the block numbers of the data pages modified by the transaction in the change bitmap; based on multiple block numbers recorded in the change bitmap, corresponding data pages are loaded from the storage unit of the primary database to obtain incremental data.

[0057] According to an embodiment of the present invention, in a distributed database system, the master database pre-builds and maintains a change bitmap data structure as a core tool for tracking data changes. The change bitmap is essentially a binary array, with each bit corresponding to a data page in the database. The "0" and "1" states indicate whether a data page has changed.

[0058] When a transaction enters the commit phase, the database engine automatically scans the data operations involved in the transaction, locates all modified data pages, and records the corresponding data page block numbers in the change bitmap. For example, if a transaction modifies data pages numbered 10, 25, and 37, bits 10, 25, and 37 in the change bitmap are marked as "1," while the corresponding bits for the remaining unchanged data pages remain "0." This real-time marking mechanism ensures that the change bitmap dynamically reflects the latest database modification status.

[0059] When performing an incremental backup, the system first retrieves the latest change bitmap. By scanning the bits in the bitmap with a value of "1," it extracts the block numbers of all modified data pages. Subsequently, based on these block numbers, it directly accesses the primary database's storage units (such as disk partitions and storage block devices) and selectively loads the corresponding data pages. Because the change bitmap precisely identifies the data modification range, the system can quickly and completely retrieve incremental data by simply reading the data pages marked as changed, without having to traverse the entire database. This change bitmap-based incremental data extraction method significantly reduces read and write operations and data transfer volume, significantly improving backup efficiency while minimizing the impact on database performance.

[0060] According to an embodiment of the present invention, the database data processing method also includes: detecting the number of block numbers recorded in the change bitmap; when it is determined that the number of block numbers is less than a preset threshold, maintaining a polling waiting state until the number of recorded block numbers increases to a preset threshold; when it is determined that the number is greater than or equal to the preset threshold, executing an operation of loading corresponding data pages from the storage unit of the main database based on multiple block numbers recorded in the change bitmap.

[0061] According to an embodiment of the present invention, in an incremental backup mechanism of a distributed database, the number of block numbers recorded in a change bitmap is monitored in real time to optimize a backup triggering strategy.

[0062] Specifically, a timer or event monitoring mechanism is used to periodically monitor the number of block numbers marked as "1" in the change bitmap. The preset threshold is an empirical value set based on factors such as database performance, network bandwidth, and workload. For example, it can be set to 1,000 block numbers. Its core purpose is to avoid wasting resources due to frequently processing small amounts of incremental data.

[0063] When the number of block numbers detected falls below a preset threshold, the system maintains a polling wait state, continuously monitoring the update of the change bitmap. During this period, if a newly submitted transaction involves a data page modification, the corresponding block number will be written to the change bitmap in real time, and the number of block numbers will gradually accumulate. By setting a reasonable polling interval (e.g., once per second), the system ensures real-time performance while avoiding excessive consumption of processor resources.

[0064] Once it is detected that the number of block numbers reaches or exceeds the preset threshold, polling is terminated immediately and the incremental data loading process is triggered. At this time, based on all the block numbers recorded in the change bitmap, the database engine will read the corresponding physical data pages from the storage unit in batches. This "threshold triggering" mechanism can merge multiple small-scale incremental changes into a batch operation, reduce the frequency of starting backup tasks, and reduce the fragmentation of read and write operations, thereby improving overall backup efficiency while avoiding the impact of frequent backups on the submission performance of business transactions. By dynamically balancing the frequency of data changes and system resource utilization, this strategy can effectively optimize the backup efficiency of distributed databases.

[0065] According to an embodiment of the present invention, based on the historical transaction processing data of the main database and the data volume of the incremental data, the backup start time of the incremental data is predicted, including: determining the estimated backup duration according to the historical backup rate of the main database and the data volume of the incremental data; performing statistics on the transaction processing volume in the historical transaction processing data to generate a transaction processing rate curve; extracting the trend change characteristics of the transaction processing volume by analyzing the transaction processing rate curve; predicting the backup start time of the incremental data according to the trend change characteristics and the estimated backup duration, wherein the backup end time determined based on the backup start time and the estimated backup duration is earlier than the target delayed reading time.

[0066] According to an embodiment of the present invention, in the incremental backup strategy of a distributed database, intelligent prediction of backup time is achieved by integrating historical data and real-time information. The historical backup records of the master database are retrieved, the data transmission volume and time consumption during each backup process are analyzed, and the historical backup rate is calculated. Combined with the current incremental data volume, a simple division operation is performed, i.e., "estimated backup duration = incremental data volume ÷ historical backup rate", to derive the estimated backup duration. This calculation process comprehensively considers key factors that affect backup speed, such as storage device read and write performance and network bandwidth, providing a quantitative basis for subsequent decision-making.

[0067] We also conduct in-depth analysis of historical transaction data, using time as a metric to compile statistics on transaction volumes over different time periods. Using data smoothing and visualization tools, we generate a transaction processing rate curve (TPS curve, meaning Transactions Per Second). This curve not only visually illustrates fluctuations in transaction volume over a day, week, or even longer periods, but also reveals patterns in the distribution of business peaks and troughs, such as high-load periods during the day on weekdays and low-load periods late at night.

[0068] Furthermore, data mining and machine learning algorithms are used to extract features from the transaction rate curve. By identifying features such as periodic fluctuations, sudden changes, and upward or downward trends in the curve, a transaction volume change model is constructed. For example, a sliding window algorithm can be used to detect the slope of recent transaction volume changes to determine whether the system load is increasing or decreasing, thereby predicting load trends over the next period of time.

[0069] Based on the above analysis results, with "the backup end time is earlier than the target delayed reading time" as the core constraint, combined with the estimated backup duration, the optimal backup start time is reversely calculated. Specifically, the trough period in the transaction processing rate curve is given priority to ensure that the impact of the backup operation on the business is minimized. At the same time, a sufficient time window is reserved to ensure that the backup task is completed before the target delayed reading time. For example, if the estimated backup duration is 30 minutes and the target delayed reading time is 2 a.m., then look for a time point with low transaction processing volume and no high load will be triggered within 30 minutes from that moment, such as 1:15 a.m., as the starting time of this incremental backup. Through this data-driven dynamic decision-making mechanism, it is possible to achieve efficient coordination between backup tasks and business operations while ensuring data timeliness.

[0070] According to an embodiment of the present invention, the backup start time of incremental data is predicted based on trend change characteristics and estimated backup duration, including: based on the trend change characteristics, identifying multiple candidate time intervals in which the transaction processing volume is lower than a preset number from the time processing sequence of the transaction processing curve; determining a target time interval that is earlier than the target delayed reading time and closest to the target delayed reading time from the multiple candidate time intervals; taking the end boundary time of the target time interval as a reference, combined with the estimated backup duration, reversely calculating the backup start time.

[0071] According to an embodiment of the present invention, the backup start time is determined through multi-dimensional screening and precise calculation based on transaction processing trend characteristics. First, a detailed analysis of the transaction processing rate curve is performed. Then, statistical methods and dynamic threshold algorithms are combined to set a reasonable transaction processing volume preset based on the fluctuation range of historical data.

[0072] For example, by calculating the average and standard deviation of transaction volume over the past week, the pre-set threshold is set at 30% of the average to accommodate natural fluctuations in business load. Subsequently, a sliding window technique is used to traverse the time series of the transaction curve, monitoring the transaction volume within each time segment in real time. If the transaction volume within a certain time interval consistently falls below the pre-set threshold, the interval is marked as a candidate time interval, and its start and end times, average transaction volume, and other information are stored in a candidate list.

[0073] After obtaining multiple candidate time intervals, the candidate list is sorted based on the dual filtering criteria of "earlier than the target delayed reading time and closest to the target." The time difference between the end boundary of each candidate time interval and the target delayed reading time is calculated, and intervals beyond the target time are eliminated. The interval with the smallest time difference is preferentially selected as the target time interval.

[0074] For example, if the target delayed reading time is 3 a.m., there are two candidate intervals - [1 a.m. - 1:30 a.m.] and [2 a.m. - 2:30 a.m.]. The latter will be selected as the target time interval to ensure that the backup task can avoid business peaks and be completed as close to the target time as possible.

[0075] After determining the target time interval, the system uses its end boundary as a benchmark, combined with the previously calculated estimated backup duration, to perform a reverse calculation. For example, if the target time interval is [2:00 AM - 2:30 AM] and the estimated backup duration is 20 minutes, the system will set the backup start time to 2:10 AM. This ensures that the backup task starts within the target time interval and ends before the end boundary, effectively reducing the impact of the backup operation on the business system while meeting the timeliness requirements of delayed data access.

[0076] According to an embodiment of the present invention, the database data processing method further includes: when the backup is completed, creating a data consistency point based on the backup end time; and associating and storing the data consistency point with the transaction log corresponding to the incremental data generated by the primary database.

[0077] According to an embodiment of the present invention, after a distributed database system completes an incremental backup, it automatically performs data consistency marking and associated storage operations to ensure the integrity and traceability of the backup data. At the moment the backup task is completed, the backup end time is accurately recorded based on the current system clock and used as a key data consistency point. This timestamp not only identifies the time boundary of the backup data but also serves as a reference point for subsequent data recovery and verification. The data consistency point is created using atomic operations, ensuring that its accuracy is not affected by concurrent transactions.

[0078] Subsequently, the transaction logs associated with this incremental data are automatically retrieved and associated. In a distributed database architecture, each transaction generates a detailed log record during execution, including information such as the operation type (addition, deletion, or modification), the data change content, and the execution timestamp. The system analyzes the transaction identifiers or timestamp ranges contained in the incremental data and filters all transaction logs involved in generating the current backup data from the log storage system. These logs record all data change operations from the last backup to the current consistency point and are the key basis for point-in-time recovery.

[0079] Finally, the data consistency points are associated with the corresponding transaction logs for storage. This association is typically stored as metadata, such as by creating an index file in a distributed file system or a mapping table in a relational database. The structural design of the associated storage must meet the requirements of efficient query and fast recovery, such as indexing transaction logs by timestamp or establishing a many-to-many association between consistency points and transaction log batches.

[0080] In this way, when data recovery is needed, the system can quickly locate the corresponding consistency point based on the specified time point, then load the associated transaction log, gradually replay the data change process, and accurately restore the database state to the target time point, thereby ensuring the data reliability and business continuity of the distributed database in the face of failures or data corruption.

[0081] According to an embodiment of the present invention, the backed-up incremental data is restored to the target delayed reading time using a delayed database, including: identifying the historical data that has been restored from the main database within the historical time period in the delayed database; copying the incremental data and transaction logs to the delayed database to cover the changed parts of the historical data; searching for the target consistency point closest to the target delayed reading time from the archive logs of the main database; restoring the incremental data in the delayed database; and replaying the transaction logs to the target consistency point to obtain the restored data.

[0082] According to an embodiment of the present invention, in a delayed recovery mechanism of a distributed database, time point synchronization between a delayed database and a main database is achieved through precise data comparison and incremental update.

[0083] First, a metadata scan is performed on the delayed database to identify historical data that has been restored from the primary database during the historical time period. This process locates all data blocks that have completed recovery operations and their corresponding time ranges by querying the delayed database's recovery record table or timestamp index. For example, by analyzing the historical recovery record table in the delayed database, the start and end times of each recovery operation, as well as the affected data page range, are obtained, thereby constructing a complete view of historical data.

[0084] Based on the above identification results, the incremental data of this backup and the associated transaction logs are copied to the delay database. During the data copying process, a block-level comparison algorithm (such as hash check) is used to accurately identify the parts of the historical data that have changed, and only these parts are overwritten and updated. This embodiment reads the latest log sequence number of the modified data page for identification, which can efficiently find the changed data of the most recent day and greatly reduce the amount of data moved. For example, the system calculates the hash value of the data page, compares the hash values of the corresponding data pages in the main database and the delay database to see if they are consistent, and only replaces the inconsistent data pages, thereby significantly reducing the amount of data transmission and storage overhead.

[0085] At the same time, the target consistency point closest to the target delayed read time is retrieved from the archive logs of the primary database. The archive logs are complete records of database transactions, containing the precise timestamp of each transaction commit. Using a binary search algorithm, the transaction record with the timestamp closest to the target delayed read time is quickly located and marked as the target consistency point. For example, if the target delayed read time is 2:00 AM, the transaction identifier in the archive log with the timestamp closest to but not later than 2:00 AM is found and used as the target consistency point.

[0086] After completing the data copy and locating the target consistency point, recovery is performed in the delayed database. First, the incremental data is loaded and written to the corresponding storage location in the delayed database. Then, transactions are replayed from the starting point corresponding to the incremental data in chronological order of the transaction log, gradually applying each transaction's changes until the target consistency point is reached. This process strictly adheres to the atomicity and isolation principles of transactions to ensure accurate data recovery. Ultimately, the data state in the delayed database is accurately restored to the target delayed read time, providing a reliable data foundation for subsequent delayed read operations.

[0087] Figure 4 A schematic diagram of the process architecture of a database data processing method according to an embodiment of the present invention is shown.

[0088] like Figure 4 As shown, the database backup and recovery process architecture adopts a modular design, with each component working together to ensure data security. Log archive server 401, as a core component, specifically stores the operation logs of the main database 404. These logs record the history of all data changes and provide a key basis for subsequent backup and recovery.

[0089] Backup estimation module 402 performs intelligent analysis based on data from primary database 404. By evaluating metrics such as log data volume and operation frequency, it dynamically determines the optimal backup strategy and resource requirements and returns the results to primary database 404. Furthermore, the output of backup estimation module 402 guides the specific operations of backup execution module 403, ensuring an efficient and economical backup process. It may analyze information such as log data volume and operation frequency to determine the appropriate backup strategy and required resources. Backup execution module 403 extracts data from primary database 404 to execute the backup task.

[0090] The recovery execution module 405 is responsible for accurately restoring the backup data to the delay database 406 when needed. The delay database 406 is specifically used to store historical data and support business scenarios with lower real-time requirements such as data analysis.

[0091] The entire architecture, from log management to backup estimation, backup execution, and final recovery, forms a complete closed loop, ensuring data security while meeting diverse business needs. This design effectively mitigates the risk of data loss while providing flexible data support for diverse business scenarios.

[0092] Figure 5 A block diagram of a database data processing device according to an embodiment of the present invention is shown.

[0093] like Figure 5 As shown, the database data processing device 500 includes a data identification module 510 , a time prediction module 520 and a data processing module 530 .

[0094] The data identification module 510 is used to perform incremental change detection on the primary database to identify incremental data that needs to be backed up.

[0095] The time prediction module 520 is used to predict the backup start time of the incremental data based on the historical transaction processing data of the primary database and the data volume of the incremental data.

[0096] The data processing module 530 is configured to perform a backup of the incremental data according to the predicted backup start time, so as to restore the backed-up incremental data to the target delayed reading time using the delayed database when the backup is completed.

[0097] According to an embodiment of the present invention, the data identification module 510 includes a change acquisition submodule and a data loading submodule.

[0098] The change acquisition submodule is used to obtain the change bitmap of the main database. During the transaction commit phase, the main database records the block number of the data page modified by the transaction into the change bitmap.

[0099] The data loading submodule is used to load corresponding data pages from the storage unit of the main database based on multiple block numbers recorded in the change bitmap to obtain incremental data.

[0100] According to an embodiment of the present invention, the database data processing device 500 further includes a quantity detection module, a polling waiting module, and an operation execution module.

[0101] The quantity detection module is used to detect the number of block numbers recorded in the change bitmap.

[0102] The polling waiting module is used to maintain the polling waiting state when it is determined that the number of block numbers is less than the preset threshold value, until the number of recorded block numbers increases to the preset threshold value.

[0103] The operation execution module is used to execute the operation of loading corresponding data pages from the storage unit of the main database based on the multiple block numbers recorded in the change bitmap when it is determined that the number is greater than or equal to a preset threshold.

[0104] According to an embodiment of the present invention, the time prediction module 520 includes a duration determination submodule, a curve generation submodule, a trend analysis submodule and a time prediction submodule.

[0105] The duration determination submodule is used to determine the estimated backup duration based on the historical backup rate of the primary database and the amount of incremental data.

[0106] The curve generation submodule is used to collect statistics on the transaction processing volume in the historical transaction processing data and generate a transaction processing rate curve.

[0107] The trend analysis submodule is used to extract the trend change characteristics of transaction processing volume by analyzing the transaction processing rate curve.

[0108] The time prediction submodule is used to predict the backup start time of incremental data based on trend change characteristics and estimated backup duration, wherein the backup end time determined based on the backup start time and estimated backup duration is earlier than the target delayed reading time.

[0109] According to an embodiment of the present invention, the time prediction submodule includes a candidate identification unit, a target determination unit and a time determination unit.

[0110] The candidate identification unit is used to identify multiple candidate time intervals in which the transaction processing volume is lower than a preset number from the time processing sequence of the transaction processing curve based on the trend change characteristics.

[0111] The target determination unit is configured to determine, from a plurality of candidate time intervals, a target time interval that is earlier than the target delayed reading time and closest to the target delayed reading time.

[0112] The time determination unit is used to reversely calculate the backup start time based on the end boundary time of the target time interval and the estimated backup duration.

[0113] According to an embodiment of the present invention, the database data processing device 500 further includes a consistency creation module and a log storage module.

[0114] The consistency creation module is used to create a data consistency point based on the backup end time when the backup is completed.

[0115] The log storage module is used to associate and store data consistency points with transaction logs corresponding to incremental data generated by the primary database.

[0116] According to an embodiment of the present invention, the data processing module 530 includes a history identification submodule, a data copy submodule, a consistency search submodule, and a data recovery submodule.

[0117] The history identification submodule is used to identify the historical data in the delayed database that has been recovered from the main database within the historical time period.

[0118] The data copy submodule is used to copy incremental data and transaction logs to the delayed database to cover the changed parts of the historical data.

[0119] The consistency search submodule is used to search the archive logs of the primary database for the target consistency point closest to the target delayed read time.

[0120] The data recovery submodule is used to recover incremental data in the delayed database and replay the transaction log to the target consistency point to obtain the recovered data.

[0121] Any number of the modules, submodules, units, and subunits according to embodiments of the present invention, or at least part of the functionality of any number of these units, can be implemented in a single module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be split into multiple modules for implementation. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware using any other reasonable method of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as a computer program module that, when executed, can perform the corresponding functionality.

[0122] For example, any number of the data identification module 510, the time prediction module 520, and the data processing module 530 can be combined into a single module / unit / sub-unit, or any one of these modules / units / sub-units can be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units can be combined with at least part of the functionality of other modules / units / sub-units and implemented in a single module / unit / sub-unit. According to an embodiment of the present invention, at least one of the data identification module 510, the time prediction module 520, and the data processing module 530 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented in hardware or firmware by any other reasonable means of circuit integration or packaging, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or any appropriate combination of any of these. Alternatively, at least one of the data identification module 510 , the time prediction module 520 and the data processing module 530 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0123] It should be noted that the database data processing device part in the embodiment of the present invention corresponds to the database data processing method part in the embodiment of the present invention. The description of the database data processing device part specifically refers to the database data processing method part and will not be repeated here.

[0124] Figure 6 A block diagram of an electronic device suitable for implementing a database data processing method according to an embodiment of the present invention is shown. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0125] like Figure 6 As shown, an electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage unit 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0126] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes the programs in ROM 602 and / or RAM 603 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute the programs stored in one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0127] According to an embodiment of the present invention, electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to bus 604. Electronic device 600 may also include one or more of the following components connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a LAN card or modem. Communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. Removable media 611, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 610 as needed, so that computer programs read from the removable media can be installed into storage section 608 as needed.

[0128] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0129] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0130] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0131] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0132] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the database data processing method provided by the embodiment of the present invention.

[0133] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the above-described systems, devices, modules, units, etc. can be implemented by computer program modules.

[0134] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0135] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages include, but are not limited to, languages such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0137] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A database data processing method, characterized in that: The method comprises: Perform incremental change detection on the master database to identify incremental data that needs to be backed up; Predicting a backup start time for the incremental data based on historical transaction processing data of the primary database and the amount of the incremental data; The incremental data is backed up according to the predicted backup start time, so that when the backup is completed, the backed up incremental data is restored to the target delayed reading time using the delayed database.

2. The method according to claim 1, characterized in that The incremental change detection on the primary database to identify incremental data that needs to be backed up includes: Obtaining a change bitmap of the primary database, wherein the primary database records the block numbers of data pages modified by the transaction in the change bitmap during the transaction commit phase; Based on the multiple block numbers recorded in the change bitmap, corresponding data pages are loaded from the storage unit of the main database to obtain the incremental data.

3. The method according to claim 2, characterized in that The method further comprises: Detecting the number of block numbers recorded in the change bitmap; When it is determined that the number of block numbers is less than the preset threshold, maintaining the polling waiting state until the number of recorded block numbers increases to the preset threshold; If it is determined that the number is greater than or equal to the preset threshold, the operation of loading corresponding data pages from the storage unit of the main database based on the multiple block numbers recorded in the change bitmap is performed.

4. The method according to claim 1, wherein The predicting of the backup start time of the incremental data based on the historical transaction processing data of the primary database and the data volume of the incremental data includes: Determining an estimated backup duration based on the historical backup rate of the primary database and the amount of the incremental data; Collecting statistics on transaction processing volumes in the historical transaction processing data to generate a transaction processing rate curve; By analyzing the transaction processing rate curve, a trend change feature of the transaction processing volume is extracted; The backup start time of the incremental data is predicted based on the trend change characteristics and the estimated backup duration, wherein the backup end time determined based on the backup start time and the estimated backup duration is earlier than the target delayed reading time.

5. The method according to claim 4, characterized in that The predicting the backup start time of the incremental data according to the trend change characteristics and the estimated backup duration includes: Based on the trend change characteristics, identifying a plurality of candidate time intervals in which the transaction processing volume is lower than a preset number from the time processing sequence of the transaction processing curve; Determine, from the plurality of candidate time intervals, a target time interval that is earlier than the target delayed reading time and closest to the target delayed reading time; The backup start time is obtained by reversely calculating the end boundary time of the target time interval in combination with the estimated backup duration.

6. The method according to claim 4, characterized in that The method further comprises: When the backup is completed, a data consistency point is created based on the backup end time; The data consistency point and the transaction log corresponding to the incremental data generated by the primary database are associated and stored.

7. The method according to claim 6, characterized in that The method of restoring the backed-up incremental data to the target delayed reading time by using the delayed database includes: identifying historical data in the delayed database that has been recovered from the primary database within a historical time period; Copying the incremental data and the transaction log to the delay database to cover the changed portion of the historical data; Searching the archive logs of the primary database for a target consistency point closest to the target delayed read time; In the delayed database, the incremental data is restored; and the transaction log is replayed to the target consistency point to obtain restored data.

8. A database data processing device, characterized in that: The device comprises: The data identification module is used to detect incremental changes in the master database to identify the incremental data that needs to be backed up; A time prediction module, configured to predict a backup start time of the incremental data based on historical transaction processing data of the primary database and the amount of the incremental data; The data processing module is used to perform a backup of the incremental data according to the predicted backup start time, so as to restore the backed up incremental data to the target delayed reading time using the delayed database when the backup is completed.

9. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Executable instructions are stored thereon, which, when executed by a processor, enable the processor to implement the method according to any one of claims 1 to 7.

Images