Data processing method, data processing device, electronic equipment and readable storage medium

Abstract

The invention discloses a data processing method, a data processing device, electronic equipment and a readable storage medium, and belongs to the technical field of computers. The method comprises the following steps: in response to triggering of a data backup event aiming at a target database, backing up corresponding first incremental data from a first log serial number to a second log serial number, the first log serial number is a page tracking starting log serial number for starting record change data corresponding to the previous data backup operation of the target database, and the second log serial number is a log serial number of a first check point after the data backup event is triggered; and in response to completion of backup of the first incremental data, backing up the Redo log.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer, in particular to a data processing method, a data processing apparatus, an electronic device and a readable storage medium. BACKGROUND

[0002] In a production environment, the reasons for data loss usually include hardware failure, software failure, misoperation, etc. To avoid data loss, a backup operation is usually taken on data. However, the related data backup methods all have the problems of missing incremental backup function or low efficiency of incremental backup, and cannot realize fast and efficient data incremental backup. SUMMARY

[0003] Embodiments of the present application provide a data processing method, a data processing apparatus, an electronic device and a readable storage medium, which can realize fast and efficient data incremental backup.

[0004] To solve the above technical problems, the present application is implemented as follows: In a first aspect, a data processing method is provided, which comprises: in response to a data backup event trigger for a target database, backing up first incremental data corresponding to a period from a first log sequence number to a second log sequence number, wherein the first log sequence number is a starting log sequence number of page tracking of recording change data corresponding to a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event trigger; and in response to completion of the first incremental data backup, backing up a Redo log.

[0005] In a second aspect, a data processing apparatus is provided, which comprises: a first backup module configured to, in response to a data backup event trigger for a target database, backup first incremental data corresponding to a period from a first log sequence number to a second log sequence number, wherein the first log sequence number is a starting log sequence number of page tracking of recording change data corresponding to a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event trigger; and a second backup module configured to, in response to completion of the first incremental data backup, backup a Redo log.

[0006] In a third aspect, an electronic device is provided, which comprises a processor and a memory, and the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the method according to the first aspect.

[0007] In a fourth aspect, a readable storage medium is provided, and the readable storage medium stores a program or instructions, and the program or instructions are executed by a processor to implement the steps of the method according to the first aspect.

[0008] In a fifth aspect, a computer program product is provided, and the computer program product comprises a computer program stored on a non-transitory computer readable storage medium, and the computer program comprises program instructions, and the program instructions, when executed by a computer, cause the computer to implement the steps of the method according to the first aspect.

[0009] In the embodiments of the present application, in response to a data backup event trigger for a target database, first incremental data corresponding to a first log sequence number to a second log sequence number is backed up, wherein the first log sequence number is a starting log sequence number of page tracking of recorded change data corresponding to a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event trigger; in response to completion of the first incremental data backup, Redo logs are backed up. In this way, by using the starting log sequence number of page tracking of recorded change data corresponding to the previous data backup operation as a starting point of the present backup, the completeness and consistency of the backup data are ensured, and only the data modified since the last full backup or incremental backup is backed up, thereby reducing the amount of data to be processed for backup, shortening the backup time. In addition, after the data page backup is completed, the Redo logs are backed up, so that the database can be restored to an arbitrary time point, and all data changes after the backup start can be completely restored during the restoration, thereby ensuring the completeness and consistency of the data during the incremental backup, and further achieving fast, efficient and reliable data incremental backup.

[0010] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0012] Figure 1 FIG. 1 shows a flowchart of a data processing method according to an exemplary embodiment of the present application; Figure 2 FIG. 2 shows a flowchart of full backup according to an exemplary embodiment of the present application; Figure 3 FIG. 3 shows a timing diagram of full backup according to an exemplary embodiment of the present application; Figure 4A flow diagram of the incremental backup provided by an example embodiment of the present application is shown. Figure 5 A timing diagram of the incremental backup provided by an example embodiment of the present application is shown. Figure 6 A storage format diagram provided by an example embodiment of the present application is shown. Figure 7 Another storage format diagram provided by an example embodiment of the present application is shown. Figure 8 A timing diagram of the data processing method provided by an example embodiment of the present application is shown. Figure 9 Another flow diagram of the data processing method provided by an example embodiment of the present application is shown. Figure 10 A timing diagram of the data processing method provided by an example embodiment of the present application is shown. Figure 11 Another flow diagram of the data processing method provided by an example embodiment of the present application is shown. Figure 12 A structural diagram of the data processing apparatus provided by an example embodiment of the present application is shown. Figure 13 A structural diagram of the electronic device provided by an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] The example embodiments will be described in detail herein with reference to the attached drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The following description of example embodiments does not represent all embodiments consistent with the present application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0014] Figure 1 A flow diagram of the data processing method provided by an example embodiment of the present application is shown. The method can be performed by an electronic device, which can include a terminal device and a network side device. In other words, the method can be performed by software or hardware installed on the electronic device. The method can include the following steps: S110: In response to a data backup event trigger for the target database, backup the corresponding first incremental data from the first log sequence number to the second log sequence number.

[0015] The first log sequence number is a starting log sequence number of a page tracking record change data of a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event is triggered. In other words, the first incremental data includes corresponding change data between the first log sequence number and the second log sequence number, and the change data includes modified data pages, and the modified data pages include at least one of the following: added data pages, deleted data pages, and updated data pages.

[0016] The log sequence number (LSN) is used to identify the position of each transaction log record. The database engine assigns a new LSN each time it writes to the Redo log. Through the LSN, the database engine can determine the order and position of the log record, so that it can accurately perform log recovery.

[0017] In an exemplary embodiment, the data backup event can include, but is not limited to, at least one of the following: (1) A timing task event, which can be automatically triggered at a preset time interval.

[0018] (2) A data operation event, which can be automatically triggered when the data in the database changes due to related operations (such as insertion, update, or deletion operations).

[0019] (3) A trigger event, which can be triggered by other related events.

[0020] (4) A manual trigger event, which is triggered by a command line, an interface button, or an operation.

[0021] The backup type of the previous data backup operation is incremental backup or full backup. It can be understood that the incremental backup provided in the present application is based on the previous full backup or incremental backup, that is, the data added or modified since the end of the previous backup is backed up. During the process of the previous data backup operation, the page tracking starting log sequence number is marked, which is used to record the starting point of the next incremental backup. The checkpoint of database backup refers to a time point when the data pages in the memory, that is, dirty pages and transaction log information, are written to the disk during the running of the database. The checkpoint can ensure that the state of the database is recorded during the backup process, so that the data can be quickly recovered when needed. In the embodiment of the present application, the second log sequence number can indicate that the data pages in the memory and the transaction log information before the first checkpoint have been written to the disk. It can be understood that during data processing, the data in the memory may become unreliable or damaged due to various reasons, such as hardware failure, sudden power failure, etc. By writing data to the disk first, it can be ensured that the data is preserved on a stable and less likely to be lost medium, so that safe backup and recovery can be performed when needed. Therefore, writing data to the disk first and then backing up can ensure the consistency and integrity of the data.

[0022] In this step, for incremental backup, the modified data pages since the last full backup or incremental backup are recorded by using the page tracking function. Therefore, in this incremental backup, the changed data pages can be backed up, realizing complete incremental backup.

[0023] The first log sequence number will be explained and described below taking the previous data backup operation as a full backup as an example. In an exemplary embodiment, the full backup can be implemented based on a cloning plug-in. The cloning plug-in is a full physical backup and recovery plug-in, which supports local or remote cloning of a database instance. Local cloning stores the backed up data on the same server as the running database instance. Remote cloning transmits data from the source database instance to the server where the receiving end database instance is located through the network. The plug-in can make the backup of the database more rapid and efficient, greatly simplifying the process of database migration, backup and recovery. The full backup function based on the cloning plug-in creates a snapshot for the full backup each time the full backup operation is started. The snapshot records all stages experienced by the full backup process and the state transition between stages, for example, as shown in Figure 2 The state transition process diagram of the full backup can include the following stages: (1) Snapshot initialization stage Snapshot initialization refers to a process of creating an initial state for backup operation by using snapshot technology in a database backup process. Database snapshot is a read-only copy of a database at a specific time point, which realizes data backup and recovery by recording data changes. Snapshot initialization provides a consistent starting point for backup operation in the entire backup process. Through snapshot initialization, it can be ensured that the database is in a stable state at the time of backup.

[0024] Before the data page backup phase, the current system log sequence number is recorded as the start backup log sequence number, which is used to identify the starting position of the backup data corresponding to this full backup. In addition, the log sequence number of the first checkpoint after snapshot initialization is recorded as the end backup log sequence number, which is used to identify the ending position of the backup data corresponding to this full backup.

[0025] (2) Data page backup phase In the data page backup phase, data pages from the start backup log sequence number to the end backup log sequence number are backed up. In addition, the following steps need to be performed before backup: Step 1: The current system log sequence number is recorded as the page tracking starting log sequence number, and the page tracking function is started. The page tracking starting log sequence number is used as the starting point of the next incremental backup, i.e. the first log sequence number. The page tracking function will track and record pages modified after the page tracking starting log sequence number, where the page is identified by its table space identifier and page identifier.

[0026] Step 2: Start the Redo log archiving function and backup data pages. After data page backup is completed, the backup completion log sequence number used to identify the consistent location information of the database instance is obtained, which is used to identify the completion of data page backup.

[0027] (3) Redo log backup phase Redo logs from the end backup log sequence number to the backup completion log sequence number are backed up.

[0028] (4) Snapshot end phase From now on, the full backup task is completed, and the snapshot information is destroyed. Although the full backup task is completed, the page tracking function is still tracking and recording data pages modified after the page tracking starting log sequence number.

[0029] The relationship between the log sequence numbers corresponding to the full backup is explained in the form of a timing diagram, as follows: Figure 3As shown, before the file backup, the start backup log sequence number and the end backup log sequence number are obtained, and then the data pages between the start backup log sequence number and the end backup log sequence number are backed up. Meanwhile, before the data page backup, the page tracking start log sequence number is obtained, which is used to indicate the start position of the data corresponding to the next incremental backup. After the data page backup is completed, the backup completion log sequence number is obtained; and then the redo log backup is performed, and the redo logs archived between the end backup log sequence number and the backup completion log sequence number are backed up.

[0030] The first log sequence number is explained below by taking the previous data backup operation as an example of incremental backup. In an exemplary embodiment, the incremental backup can be implemented based on a clone plug-in. The incremental backup function based on the clone plug-in creates a snapshot for the incremental backup each time the incremental backup operation is started, and the snapshot records all stages experienced by the incremental backup process and the state transitions between the stages, for example, Figure 4 As shown, the state transition flowchart for the incremental backup can include the following stages: (1) Snapshot initialization stage The related content about the snapshot initialization stage can be referred to the related description in the previous embodiment, which will not be repeated here.

[0031] (2) Data page backup stage Before the backup, the page tracking start log sequence number of the last backup task marker is obtained and used as the start log sequence number of the modified data page, i.e., the start backup log sequence number, which is used to indicate the start position of the backup data corresponding to the incremental backup. In addition, the log sequence number of the first checkpoint after the snapshot initialization is recorded as the end backup log sequence number, which is used to indicate the end position of the backup data corresponding to the incremental backup.

[0032] In the data page backup stage, the modified data pages between the start backup log sequence number and the end backup log sequence number are backed up. Before the backup, the following steps also need to be performed: Step 1: The log sequence number of the system at the current time is used as the new page tracking start log sequence number, and the page tracking function is started. The page tracking start log sequence number is used as the start point of the next incremental backup, i.e., the first log sequence number, and the page tracking function tracks and records the pages modified after the page tracking start log sequence number, wherein the pages are identified by the table space identifier of the table space to which the pages belong and the page identifier of the pages themselves.

[0033] Step 2: Start the Redo log archiving function, and backup the modified data page. After the data page backup is completed, a backup completion log sequence number used to identify the consistent location information of the database instance is obtained, and the backup completion log sequence number is used to identify the completion of the data page backup.

[0034] (3) Redo log backup phase Backup the Redo log between the end backup log sequence number and the backup completion log sequence number.

[0035] (4) Snapshot end phase From this point, the incremental backup ends, and enters the "snapshot end" phase, and the snapshot information is destroyed. Although the incremental backup task ends, the page tracking function still tracks and records the data page modified after the page tracking start log sequence number marked in this backup.

[0036] The following illustrates the relationship between the log sequence numbers of the incremental backup and the full backup in the form of a timing diagram, as shown in the example of FIG. 6. Figure 5 As shown in FIG. 6, for the current incremental backup, the start backup log sequence number is the page tracking start log sequence number marked in the last full backup or incremental backup; then a new page tracking start log sequence number is marked as the start backup log sequence number of the next incremental backup, and the log sequence number of the first checkpoint after the snapshot is initialized is taken as the end backup log sequence number of the current incremental backup, then the data page between the start backup log sequence number and the end backup log sequence number is backed up, after the data page backup is completed, the backup completion log sequence number is obtained, and then the Redo log backup is performed, and the Redo log archived between the end backup log sequence number and the backup completion log sequence number is backed up.

[0037] Therefore, in the embodiment of the present application, the start backup log sequence number corresponding to the current backup operation, that is, the first log sequence number, is the page tracking start log sequence number marked in the last incremental backup or full backup. In an example embodiment, the page tracking start log sequence number marked in the last incremental backup or full backup can be transmitted through the command of the incremental backup.

[0038] S120: Backup the Redo log in response to the completion of the first incremental data backup.

[0039] It can be understood that the incremental backup in the embodiment of the present application can include data copying and Redo log backup, and then when the first data modified page backup is completed, it indicates that the data copying part corresponding to the current incremental backup has been completed, and therefore the Redo log needs to be backed up.

[0040] In an example embodiment, the above S120 can include the following steps: S122: Obtain a third log sequence number, wherein the third log sequence number is a log sequence number identifying completion of the first incremental data backup.

[0041] S124: Archive Redo logs corresponding to a period from the second log sequence number to the third log sequence number.

[0042] The Redo log archive refers to copying and saving the contents of the Redo Log to an archive file after performing a database backup, to ensure that no important log information is lost during the backup process. The Redo log records all modification operations of the database, including insertion, update, and deletion, etc. After the backup is completed, if the database fails, relying solely on the backup file may not be able to recover to the latest state before the failure occurred. By archiving the Redo log, all data changes after the backup can be recorded, so that these logs can be reproduced when recovering, ensuring the integrity and consistency of the data.

[0043] S126: Backup the archived Redo logs.

[0044] It can be understood that during the incremental backup process, new data changes may occur in the database, so starting to record the Redo log from the second log sequence number can capture all modification operations on the database from the second log sequence number to the third log sequence number. These logs record the incremental changes of the data, ensuring that these changes can be restored when recovering. If the Redo log is not recorded from the second log sequence number, the changes during the incremental backup process will not be recorded and restored, which may result in data loss or data inconsistency. In the embodiments of the present application, there is a certain time interval between the second log sequence number and the third log sequence number, and during the data backup process, the modified pages from the first log sequence number to the second log sequence number are backed up. If any modification operation generated between the second log sequence number and the third log sequence number is not recorded, when a failure occurs, the database may not be completely restored to the state before the failure occurred, but only to the state corresponding to the second log sequence number, i.e., the state of the data that has been backed up, resulting in loss of data between the second log sequence number and the third log sequence number.

[0045] In the embodiment of the present application, in response to a data backup event trigger for a target database, first incremental data corresponding to a first log sequence number to a second log sequence number is backed up, wherein the first log sequence number is a starting log sequence number of page tracking of recorded change data corresponding to a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event trigger; in response to completion of the first incremental data backup, Redo logs are backed up. Thus, by using the starting log sequence number of page tracking of recorded change data corresponding to the previous data backup operation as a starting point of the present backup, the integrity and consistency of the backup data are ensured, and only the data modified since the last full backup or incremental backup is backed up, thereby reducing the amount of data to be processed for backup, shortening the backup time. In addition, after the data page backup is completed, the Redo logs are backed up, so that the database can be restored to an arbitrary time point, and all data changes after the backup start can be completely restored during the restoration, thereby ensuring the integrity and consistency of the data during the incremental backup, and thereby realizing fast, efficient and reliable incremental data backup.

[0046] In addition, the data processing method provided in the embodiment of the present application can be implemented based on a MySQL clone plug-in, and can be automatically adapted with the iteration of the MySQL version, thereby reducing the operation and maintenance cost.

[0047] In an exemplary embodiment, the first incremental data corresponding to the first log sequence number to the second log sequence number includes: S112: dividing the first incremental data into at least one first data modification page according to a preset page size.

[0048] S114: dividing the at least one first data modification page into at least one data storage segment, wherein each data storage segment sequentially includes an index page and at least one first data modification page, and the index page in each data storage segment includes a segment end identifier and a page identifier corresponding to each first data modification page, and the segment end identifier is used to identify whether the data storage segment is the last data storage segment.

[0049] S116: backing up the first data modification page included in each data storage segment to a corresponding table space.

[0050] The table space is an incremental backup table space.

[0051] It can be understood that the first incremental data is divided into a plurality of first data modification pages according to a preset page size, and at least one first data modification page is divided into a data storage segment. Exemplarily, as shown in FIG. 4, the first incremental data is divided into a plurality of first data modification pages 401, 402, 403, 404 and 405 according to a preset page size, and the at least one first data modification page is divided into a data storage segment 406. Figure 6The diagram illustrates the storage format of incremental data. The first page of each data storage segment is an index page, used to record the page identifier of the data pages stored in that segment, and a segment end identifier to indicate whether the data storage segment is the last data storage segment. The segment end identifier can be "IDTFY" or "idtfy". If it is "IDTFY", it indicates that the segment is the last segment; if it is "idtfy", it indicates that the segment is not the last segment. Optionally, each page identifier and segment end identifier can occupy 4 bytes.

[0052] In another exemplary embodiment, such as Figure 7 As shown, each data storage segment also includes an end page, used to identify the last data page of the segment. This page stores invalid data and is used to identify the last page corresponding to the data storage segment. For example, the invalid data can be "0xFFFFFFFF".

[0053] Understandably, in related technologies, during full data backup, if changed data pages in the source database need to be directly sent to the target database, both the source and target databases must be online in real time. Furthermore, to achieve a secondary incremental backup, the entire database needs to be backed up again, which impacts database performance and storage space. Additionally, during data recovery, it's necessary to restore to the target database at different points in time based on actual business needs, but existing mechanisms struggle to support efficient data recovery at any given moment. Therefore, to support incremental database backup and recovery capabilities, this application provides an incremental file storage method: storing the first incremental data in segments, using an index page + data page format, thereby storing the data pages generated by the incremental database backup. This effectively supports incremental database backup capabilities. Furthermore, unlike traditional logical backups, this application can directly perform physical backup storage of data pages added since the last backup, effectively improving backup efficiency. Simultaneously, in subsequent recovery processes, recovery is performed in segments, further improving recovery efficiency.

[0054] In an exemplary embodiment, before backing up the first incremental data corresponding to the period from the first log sequence number to the second log sequence number, the method further includes: obtaining a fourth log sequence number, wherein the fourth log sequence number is the starting log sequence number for page tracking to start recording changed data corresponding to the current data backup operation of the target database, and the fourth log sequence number is used to indicate the starting position for obtaining the second incremental data corresponding to the next incremental backup of the target database; starting from the fourth log sequence number, initiating a page tracking function, wherein the page tracking function is used to track and record the second data modification page generated after the fourth log sequence number, and the second incremental data includes the second data modification page.

[0055] It can be understood that the fourth log sequence number is acquired each time the backup operation is performed, as a starting point for the next incremental backup, and the page tracking function is started from the fourth log sequence number, and the page tracking function tracks and records the data pages changed after the fourth log sequence number. After the current backup task is completed, the page tracking function still tracks and records the second data modification page modified after the fourth log sequence number, which will be used for the next incremental backup task.

[0056] The following describes the method for implementing the incremental backup by taking the example of the database backup. Figure 8 The time sequence of each log sequence number in the embodiment of the present application is schematically described as shown in the following table: Figure 8 For the current incremental backup, the first log sequence number is the page tracking starting log sequence number of the last full backup or incremental backup, and a new page tracking starting log sequence number, i.e., the fourth log sequence number, is marked as the first log sequence number of the next incremental backup, and the log sequence number of the first checkpoint after the snapshot is initialized is taken as the second log sequence number of the current incremental backup, and then the first data modification page between the first log sequence number and the second log sequence number is backed up, and after the first data modification page backup is completed, the backup completion log sequence number, i.e., the third log sequence number, is acquired, and then the Redo log archiving and backup are performed, i.e., the Redo log archived between the second log sequence number and the third log sequence number is backed up.

[0057] In an exemplary embodiment, after the backup from the first log sequence number to the corresponding first incremental data between the second log sequence number, the method further comprises: in response to the target database starting, applying the first data modification page included in each of the data storage segments to the full backup table space.

[0058] It can be understood that the current backup is an incremental backup, and only the data pages changed since the last backup are recorded, without containing the complete database. The full backup backs up all the data of the entire database or table space, and when the data needs to be restored, the last full backup and all the incremental backups need to be combined to restore to a complete state. Therefore, applying the first data modification page to the table space corresponding to the full backup can further ensure the integrity and reliability of the data on the basis of the efficiency of the incremental backup, the more flexible and more resource-saving backup strategy. In addition, the application operation is performed after the target database starts each time, without additional configuration, and the resource occupation is less, reducing the performance interference during the normal operation of the database. Exemplarily, when the target database is started, a synchronization option is added, and the data directory of the incremental backup is specified, for applying the incremental backup data to the data directory of the full backup, and automatically exiting after applying all the incremental backup data.

[0059] Further, in another example embodiment, the applying the first data modification page to the table space corresponding to the full backup can include the following steps: Step 1: obtaining a target data storage segment, wherein the target data storage segment is the data storage segment currently traversed.

[0060] Step 2: determining a table space name of a table space corresponding to the first data modification page currently traversed, which is included in the target data storage segment.

[0061] In an example embodiment, each first data modification page can be named in the format of "table space name.delta".

[0062] Step 3: obtaining a target file corresponding to the table space name from a file directory corresponding to the full backup table space.

[0063] That is, obtaining the table space name from the format of "table space name.delta", and then finding the target file corresponding to the table space name in the file directory corresponding to the full backup table space.

[0064] Step 4: in response to obtaining a target data page matching the page identifier of the first data modification page currently traversed from the target file, determining the offset of the first data modification page currently traversed in the full backup table space according to the page identifier and the page size of the first data modification page currently traversed.

[0065] Step 5: determining a modification position in the full backup table space according to the offset, and writing the first data modification page currently traversed into the modification position. Regarding the above steps 4 and 5, it can be understood that the target file includes a plurality of target data pages, each target data page corresponds to a page identifier, and the target data page matching the page identifier of the first data modification page currently traversed is the data page of the full backup corresponding to the first data modification page currently traversed. Then, according to the page identifier and the page size of the first data modification page currently traversed, the offset of the page in the full backup table space is calculated, and then the data page after the offset is overwritten.

[0066] Step 6: traversing the next first data modification page and returning to execute the step of determining the table space name of the table space corresponding to the first data modification page currently traversed, which is included in the target data storage segment, until all the first data modification pages are traversed and the step of obtaining the target data storage segment is executed.

[0067] In the embodiments of the present application, the data modification page corresponding to the segment in the incremental space is backed up into the full space in units of segments, which can effectively improve the access efficiency of the data modification page.

[0068] The following is a detailed description of the above-mentioned embodiments. As shown in the flow chart of FIG. 9, the method comprises the following steps: Figure 9 Figure 9 S910: reading the index page of each data storage segment.

[0069] S920: determining whether the data storage segment is the last segment.

[0070] If yes, go to S930, otherwise go to S940.

[0071] S930: setting the end marker last_seg = true.

[0072] The end marker last_seg can be initialized as false.

[0073] It can be understood that the first 4 bytes of the index page are read. If the character is "IDTFY", it means that the segment is the last segment, and last_seg is set to true. If the character is "idtfy", last_seg remains false.

[0074] S940: reading the page identifier included in the index page of the data storage segment.

[0075] It can be understood that the second 4 bytes of the index page are read to obtain the page identifier PageID. The data page corresponding to the PageID is the second page in the segment. In this way, the positions of the data pages corresponding to other PageIDs in the segment can be obtained.

[0076] S950: determining whether the page identifier is the last page identifier.

[0077] If yes, go to S960, otherwise go to S970.

[0078] S960: determining whether last_seg is equal to true.

[0079] If yes, end, otherwise go to S910.

[0080] ​​Understandably, the process involves determining whether the page identifier is the last page identifier of the index page. The last PageID is an invalid page identifier, and the actual stored data is "0xFFFFFFFF". If the page identifier is not the last page identifier of the index page, the data pages in the segment are written to the corresponding full backup tablespace. Otherwise, it means that all data pages of the segment have been written to the full backup tablespace, and it is determined whether last_seg is true. If not, it proceeds to S910 to read and apply the next segment. Otherwise, it means that the data modification pages corresponding to the incremental backup have been applied.

[0081] S970: Read the page data corresponding to the page identifier and write it to the full backup tablespace.

[0082] In one exemplary embodiment, after the backup redo log, the method further includes: in response to a data recovery event triggered for the target database, obtaining the data modification page corresponding to the period from the end time of the last incremental backup to the target time as needed to restore to the target time; restoring the data modification page, and restoring the redo log corresponding to the last incremental backup.

[0083] It is understandable that, such as Figure 10 As shown, each incremental backup operation consists of two parts: data page backup and redo log backup. The second incremental backup operation's data page backup 2 backs up the newly added / modified data page 1 from the first incremental backup operation; the third incremental backup operation's data page backup 3 backs up the newly added / modified data page 2 from the second incremental backup operation. Since each incremental backup's data pages already contain the redo log data from the previous backup, backups prior to the last incremental backup only need to save the backed-up data pages and redo logs. When restoring data pages, all incremental backup data pages are restored; when restoring redo logs, only the redo log from the last incremental backup needs to be restored. In other words, multiple incremental data recovery operations only require restoring the data pages from multiple backups and the redo log from the last backup, improving recovery efficiency.

[0084] In addition, in this embodiment of the application, for incremental database recovery, the incremental data page application mechanism can effectively improve recovery efficiency; and through the MySQL crash recovery mechanism, the backup redo logs are used to restore the database to a consistent state.

[0085] In one exemplary embodiment, the method further includes: updating a memory data object during the backup process, wherein the memory data object includes at least one of the following: (1) Backup type, used to identify whether this backup operation is a full backup or an incremental backup; (2) backup status, used to identify whether the current backup operation is successful; (3) backup status corresponding to the explanatory information; (4) backup start time; (5) backup end time; (6) backup source data, used to identify the source data corresponding to the first incremental data; (7) backup directory, used to identify the storage directory of the first incremental data; (8) start log sequence number, used to identify the starting log sequence number corresponding to the current backup operation; (9) end log sequence number, used to identify the end log sequence number corresponding to the current backup operation; (10) page tracking starting log sequence number, used to identify the starting position corresponding to the next backup operation.

[0086] Among them, the start log sequence number can be the first log sequence number, the end log sequence number can be the second log sequence number, and the page tracking starting log sequence number can be the fourth log sequence number in the foregoing embodiments.

[0087] In another exemplary embodiment, the memory data object further includes but is not limited to backup completion log sequence number, object identification, user operation identification, wherein the backup completion log sequence number can be the third log sequence number in the foregoing embodiments.

[0088] It can be understood that the execution information of the traditional backup tool is recorded in the file of the backup directory, that is, the start log sequence number, the page tracking starting log sequence number, the end log sequence number, and other basic information are recorded in the file. After each backup is completed, although the terminal can print the success or failure information in time, the operation and maintenance personnel need to record it, otherwise the subsequent needs to be determined by checking the backup file one by one to determine the execution result of each backup.

[0089] In the embodiment of the application, by creating a memory data object, the persistent storage of information in the incremental backup process can be realized, so that the multiple backup information can be queried by the sql command, and the operation and maintenance efficiency is improved. Exemplarily, the memory data object is shown in Table 1.

[0090] Table 1

[0091] The above embodiments will be described illustratively as follows. Figure 11 As shown in FIG. 1, the method comprises the following steps: Figure 11 ​S1110: create a memory data object, and record a backup start time, a start log sequence number, a backup type, and a backup directory.

[0092] S1120: wait for backup start.

[0093] If the backup has not started within the first preset time period, it is determined that the backup fails, and the process proceeds to S1160; otherwise, the process proceeds to S1130.

[0094] S1130: update the memory data object, and record a page tracking start log sequence number.

[0095] S1140: wait for backup end.

[0096] If the backup has not ended within the second preset time period, it is determined that the backup fails, and the process proceeds to S1160; otherwise, the process proceeds to S1150.

[0097] S1150: update the memory data object, and record an end log sequence number and a backup end time.

[0098] S1160: update the memory data object, and record a backup status, explanation information corresponding to the backup status, and a backup end time.

[0099] In the embodiments of the present application, an operation record of each backup can be obtained by an operation and maintenance personnel, and a view can be created based on the operation record to realize customized query of backup information of interest. Meanwhile, a complete link of one full backup + N incremental backups and a storage path of each backup can be determined through one SQL query command, and thus when data recovery is needed, the data can be recovered one by one according to the query result.

[0100] In addition, if multiple concurrent backup tasks are executed simultaneously, a memory data object can be created for each backup task, different tasks are isolated from each other through different snapshots, and do not affect each other, resource isolation of different data sources is realized, a single backup node can support data backup of multiple database instances, and thus resource utilization is improved.

[0101] Figure 12 A structure schematic diagram of a data processing apparatus provided by one embodiment of the present application is shown in FIG. 12. Figure 12 As shown in FIG. 12, the data processing apparatus 1200 can include a first backup module 1210 and a second backup module 1220.

[0102] In this embodiment, the first backup module 1210 is configured to, in response to a data backup event trigger for a target database, backup first incremental data corresponding to a first log sequence number to a second log sequence number, wherein the first log sequence number is a starting log sequence number of page tracking of a first record change data corresponding to a previous data backup operation of the target database, and the second log sequence number is a log sequence number of a first checkpoint after the data backup event trigger; and the second backup module 1220 is configured to, in response to completion of the first incremental data backup, backup Redo logs.

[0103] In an example embodiment, the first backup module 1210 is specifically configured to: divide the first incremental data into at least one first data modification page according to a preset page size; divide the at least one first data modification page into at least one data storage segment, wherein each data storage segment sequentially includes an index page and at least one first data modification page, the index page in each data storage segment includes a segment end identifier and a page identifier corresponding to each first data modification page, and the segment end identifier is used to identify whether the data storage segment is a last data storage segment; and backup the first data modification page included in each data storage segment to a corresponding table space.

[0104] In an example embodiment, the first processing module is further configured to, in response to starting of the target database, apply the first data modification page included in each data storage segment to a full backup table space.

[0105] In an example embodiment, the first processing module is specifically configured to: obtain a target data storage segment, wherein the target data storage segment is a currently traversed data storage segment; determine a table space name of a table space corresponding to the first data modification page currently traversed by the target data storage segment; obtain a target file corresponding to the table space name from a file directory corresponding to the full backup table space; in response to obtaining a target data page matching a page identifier of the first data modification page currently traversed from the target file, determine an offset of the first data modification page currently traversed in the full backup table space according to the page identifier and a page size of the first data modification page currently traversed; determine a modification position in the full backup table space according to the offset, and write the first data modification page currently traversed to the modification position; traverse a next first data modification page and return to execute the step of determining the table space name of the table space corresponding to the first data modification page currently traversed by the target data storage segment, until all the first data modification pages are traversed completely and the step of obtaining the target data storage segment is returned to be executed.

[0106] In an example embodiment, the second backup module 1220 is specifically configured to: acquire a third log sequence number, wherein the third log sequence number is a log sequence number identifying completion of the first incremental data backup; archive Redo logs corresponding to a period from the second log sequence number to the third log sequence number; and backup the archived Redo logs.

[0107] In an example embodiment, the method further includes a second processing module configured to: acquire a fourth log sequence number, wherein the fourth log sequence number is a page tracking start log sequence number corresponding to a current data backup operation of the target database that starts to record changed data, and the fourth log sequence number is used to indicate a start position of second incremental data corresponding to a next incremental backup of the target database; and start a page tracking function from the fourth log sequence number, wherein the page tracking function is used to track and record second data modification pages generated after the fourth log sequence number, and the second incremental data includes the second data modification pages.

[0108] In an example embodiment, the method further includes a third processing module configured to: in response to a data recovery event trigger of the target database, acquire data modification pages corresponding to a period from a last incremental backup end time to a target time to which recovery is needed according to the target time; and recover the data modification pages and recover redo logs corresponding to the last incremental backup.

[0109] In an example embodiment, the method further includes a fourth processing module configured to: update an in-memory data object during a backup process, wherein the in-memory data object includes at least one of: a backup type used to identify whether a current backup operation is a full backup or an incremental backup; a backup state used to identify whether the current backup operation is successful; description information corresponding to the backup state; a backup start time; a backup end time; backup source data used to identify source data corresponding to the first incremental data; a backup directory used to identify a storage directory of the first incremental data; a start log sequence number used to identify a start log sequence number corresponding to the current backup operation; an end log sequence number used to identify an end log sequence number corresponding to the current backup operation; and a page tracking start log sequence number used to identify a start position corresponding to a next backup operation.

[0110] The data processing apparatus provided in the embodiments of the present application can achieve Figure 1 The processes implemented in the method embodiments shown above will not be described again here to avoid repetition.

[0111] The data processing apparatus in the embodiments of the present application can be an apparatus, a component in an electronic device, an integrated circuit, or a chip. The embodiments of the present application are not specifically limited.

[0112] The data processing apparatus in the embodiments of the present application can be an apparatus with an operating system. The operating system can be an Android operating system, an ios operating system, or other possible operating systems, which are not limited in the embodiments of the present application.

[0113] As shown in Figure 13 The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned Figures 1 to 11 The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned

[0114] The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned Figures 1 to 11 The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned

[0115] The processor is the processor in the terminal in the above-mentioned embodiments. The readable storage medium includes a computer readable storage medium, such as a computer readable only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, etc. In some examples, the readable storage medium can be a non-transitory readable storage medium.

[0116] The embodiments of the present application also provide a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to realize the above-mentioned Figures 1 to 11 The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned

[0117] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0118] The embodiments of the present application also provide a computer program / program product, which includes a computer program stored on a non-transitory computer readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the above-mentioned Figures 1 to 11 The embodiments shown in the embodiments of the present application also provide an electronic device 1300, which includes a processor 1310 and a memory 1320, and the memory 1320 stores programs or instructions executable on the processor 1310. When the programs or instructions are executed by the processor 1310, the above-mentioned

[0119] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, it is to be understood that the method and apparatus of the present application can be carried out by specific hardware, software, or a combination thereof, and that the method and apparatus of the present application can be implemented in a computer program product that is executable by a computer or network-side device. In addition, it should be pointed out that the scope of the method and apparatus of the present application is not limited to the order of performing the functions as shown or discussed, but can also include performing the functions in a substantially simultaneous manner or in a reverse order, for example, the described method can be performed in an order different from that described, and various steps can be added, omitted, or combined. In addition, features described with reference to certain examples can be combined in other examples.

[0120] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of computer software products and general hardware platforms, and of course, can also be realized by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disc, optical disc, etc.), and includes a plurality of instructions for making a terminal or network-side device execute the method described in each embodiment of the present application.

[0121] The embodiments of the present application are described above in combination with the drawings, but the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are only illustrative, but not restrictive. Those skilled in the art can make many forms of embodiments under the inspiration of the present application without departing from the scope of the present application and the scope protected by the claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to a data backup event triggered for the target database, the backup process backs up the first incremental data corresponding to the interval from the first log sequence number to the second log sequence number. The first log sequence number is the starting log sequence number of the page tracking for recording changed data corresponding to the previous data backup operation of the target database, and the second log sequence number is the log sequence number of the first checkpoint after the data backup event is triggered. In response to the completion of the first incremental data backup, the Redo log is backed up.

2. The method according to claim 1, characterized in that, The backup of the first incremental data corresponding to the interval from the first log sequence number to the second log sequence number includes: The first incremental data is divided into at least one first data modification page according to a preset page size; The at least one first data modification page is divided into at least one data storage segment, wherein each data storage segment sequentially includes an index page and at least one first data modification page, and the index page in each data storage segment includes a segment end identifier and a page identifier corresponding to each first data modification page, wherein the segment end identifier is used to identify whether the data storage segment is the last data storage segment; The first data modification page included in each of the data storage segments is backed up to the corresponding tablespace.

3. The method according to claim 2, characterized in that, After backing up the first incremental data corresponding to the interval from the first log sequence number to the second log sequence number, the method further includes: In response to the startup of the target database, the first data modification page included in each of the data storage segments is applied to the full backup tablespace.

4. The method according to claim 3, characterized in that, The step of applying the first data modification page included in each of the data storage segments to the tablespace corresponding to the full backup includes: Obtain the target data storage segment, wherein the target data storage segment is the data storage segment currently being traversed; Determine the tablespace name of the tablespace corresponding to the first data modification page currently being traversed, including the target data storage segment; Obtain the target file corresponding to the tablespace name from the file directory corresponding to the full backup tablespace; In response to obtaining a target data page from the target file that matches the page identifier of the currently traversed first data modification page, the offset of the currently traversed first data modification page in the full backup tablespace is determined based on the page identifier and the page size of the currently traversed first data modification page. Based on the offset, determine the modification location from the full backup tablespace, and write the currently traversed first data modification page to the modification location; The process involves iterating through the next first data modification page and returning to the step of determining the tablespace name of the tablespace corresponding to the currently traversed first data modification page, including the target data storage segment, until all first data modification pages have been traversed. The process then returns to the step of obtaining the target data storage segment.

5. The method according to claim 1, characterized in that, The backup redo logs include: Obtain the third log sequence number, wherein the third log sequence number is the log sequence number that identifies the completion of the first incremental data backup; Archive the Redo logs corresponding to the sequence number from the second log sequence number to the third log sequence number; Backup the archived Redo logs.

6. The method according to claim 1, characterized in that, Before backing up the first incremental data corresponding to the interval from the first log sequence number to the second log sequence number, the method further includes: Obtain the fourth log sequence number, wherein the fourth log sequence number is the starting log sequence number of the page tracking for recording changed data corresponding to the current data backup operation of the target database, and the fourth log sequence number is used to indicate the starting position for obtaining the second incremental data corresponding to the next incremental backup of the target database. Starting from the fourth log sequence number, the page tracking function is initiated, wherein the page tracking function is used to track and record the second data modification page generated after the fourth log sequence number, and the second incremental data includes the second data modification page.

7. The method according to any one of claims 1-6, characterized in that, Following the backup of the redo logs, the method further includes: In response to a data recovery event triggered for the target database, the data modification pages corresponding to the end time of the last incremental backup and the target time are retrieved as needed to restore to the target time. Restore the data modification page and restore the redo log corresponding to the last incremental backup.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: During the backup process, the memory data object is updated, wherein the memory data object includes at least one of the following: Backup type, used to identify whether this backup operation is a full backup or an incremental backup; Backup status is used to indicate whether the backup operation was successful; Explanatory information corresponding to the backup status; Backup start time; Backup end time; Backup source data, used to identify the source data corresponding to the first incremental data; Backup directory, used to identify the storage directory of the first incremental data; The start log sequence number is used to identify the starting log sequence number corresponding to this backup operation. End log sequence number, used to identify the end log sequence number corresponding to this backup operation; The page tracks the starting log sequence number, which is used to identify the starting position of the next backup operation.

9. A data processing apparatus, characterized in that, include: The first backup module is used to back up the first incremental data corresponding to the period from the first log sequence number to the second log sequence number in response to the triggering of a data backup event for the target database. The first log sequence number is the starting log sequence number of the page tracking for opening the record of changed data corresponding to the previous data backup operation of the target database, and the second log sequence number is the log sequence number of the first checkpoint after the data backup event is triggered. The second backup module is used to back up the redo log in response to the completion of the first incremental data backup.

10. An electronic device, characterized in that, include: A processor and a memory, wherein the memory stores a program or instructions executable on the processor, the program or instructions, when executed by the processor, implement the method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

Images