A method, apparatus, device, and storage medium for determining breakpoints in log files.

Abstract

This invention discloses a method, apparatus, device, and storage medium for determining breakpoints in log files. The method includes: upon detecting the start of a backup service, acquiring a pre-recorded current log breakpoint; reading transaction data from the current log breakpoint in the log file and performing a data backup operation; during the backup operation, updating a pre-created list of incomplete backup transactions and a list of invalid transactions; and when breakpoint determination conditions are met, determining and saving the next log breakpoint corresponding to the next start of the backup service based on the list of incomplete backup transactions and the list of invalid transactions. Considering that in the prior art, after a backup service restart, the log file corresponding to the breakpoint may have been cleaned up and cannot be backed up normally, this solution addresses this problem by ignoring invalid long transactions when determining breakpoints based on an invalid transaction list, thereby preventing the backup service from failing to find the log file corresponding to the breakpoint after a restart and ensuring normal backup after restart.

Description

Technical Field

[0001] This invention relates to the field of database backup technology, and in particular to a method, apparatus, device, and storage medium for determining breakpoints in log files. Background Technology

[0002] Data backup is the foundation of disaster recovery. To ensure data security, data from the source database is backed up to a backup database in real time. Data backup is performed by the source database through analysis of transaction data. To ensure data consistency between the backup and source databases, no transaction data can be missed during the analysis process.

[0003] Considering that the backup process may be paused, exited, or restarted, when the backup service restarts, it needs to start the analysis again from the transactions that were not completed before the last exit, so that no transaction data is missed. Therefore, during the analysis process, the position of the start information of the transaction being analyzed in the log file needs to be recorded as a breakpoint and saved. This breakpoint can serve as the starting point for analysis after a restart.

[0004] In real-world scenarios, we often encounter situations where there are incomplete transactions, causing the log file corresponding to the breakpoint to be in an earlier time period. Due to storage space limitations, log files are often cleaned up, and the file corresponding to the breakpoint may have been deleted, making it impossible to back up properly after a restart. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for determining breakpoints in log files, in order to solve the problem that backup services cannot perform normal backups after restarting because the log files corresponding to the breakpoints cannot be found.

[0006] According to one aspect of the present invention, a method for determining breakpoints in a log file is provided, comprising:

[0007] When the backup service is detected to be started, the pre-recorded current log breakpoint is obtained, transaction data is read from the current log breakpoint in the log file, and data backup operation is performed.

[0008] During the backup operation, update the pre-created list of incomplete backup transactions and the list of invalid transactions;

[0009] When the breakpoint determination conditions are met, the next log breakpoint corresponding to the next startup of the backup service is determined and saved based on the list of incomplete backup transactions and the list of invalid transactions.

[0010] According to another aspect of the present invention, a breakpoint determination device for log files is provided, comprising:

[0011] The data backup operation module is used to obtain the pre-recorded current log breakpoint after the backup service is detected to be started, read transaction data from the current log breakpoint in the log file and perform data backup operation;

[0012] The transaction list update module is used to update the pre-created list of incomplete backup transactions and the list of invalid transactions during the backup operation.

[0013] The breakpoint determination module is used to determine and save the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions when the breakpoint determination conditions are met.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the log file breakpoint determination method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log file breakpoint determination method according to any embodiment of the present invention.

[0019] This invention discloses a method, apparatus, device, and storage medium for determining breakpoints in log files. The method includes: upon detecting the start of a backup service, acquiring a pre-recorded current log breakpoint; reading transaction data from the current log breakpoint in the log file and performing a data backup operation; during the backup operation, updating a pre-created list of incomplete backup transactions and a list of invalid transactions; and when breakpoint determination conditions are met, determining and saving the next log breakpoint corresponding to the next start of the backup service based on the list of incomplete backup transactions and the list of invalid transactions. Considering that in the prior art, after a backup service restart, the log file corresponding to the breakpoint may have been cleaned up and cannot be backed up normally, this solution addresses this problem by ignoring invalid long transactions when determining breakpoints based on an invalid transaction list, thereby preventing the backup service from failing to find the log file corresponding to the breakpoint after a restart and ensuring normal backup after restart.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a method for determining breakpoints in a log file according to Embodiment 1 of the present invention;

[0023] Figure 2 This is a flowchart of a method for determining breakpoints in a log file, provided in Embodiment 2 of the present invention;

[0024] Figure 3 A flowchart illustrating the steps after restarting the Oracle database backup service;

[0025] Figure 4 This is a schematic diagram of the structure of a log file breakpoint determination device provided in Embodiment 3 of the present invention;

[0026] Figure 5 A schematic diagram of an electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0027] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "target," "original," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] It's important to note that to ensure the standby database remains consistent with the source database, when backing up data from the source database to the standby, it's necessary to be able to fully read all transaction data generated in the source database since the initial synchronization point, correctly parse and reverse-engineer it into database operation statements, and then re-execute them in the standby. No transaction data can be missed during this process; otherwise, inconsistency between the standby and source databases will occur. Considering that the backup service may pause, exit, or restart midway, upon restarting, it needs to start the analysis again from those transactions that were not completed before the last exit, ensuring no transaction data is missed. Therefore, during the backup service, the position of the transaction start information in the log file of the transaction being analyzed needs to be recorded as a breakpoint and saved to disk for easy re-analysis after the next backup service interruption. However, it's common to encounter situations where some transactions have been running for a long time without ending, even after multiple log file switches. In such cases, when the backup service restarts and needs to re-analyze the transaction, it needs to start the analysis from a very early log file. However, due to disk capacity limitations, these earlier log files often no longer exist, making it impossible to analyze the transaction and causing an error that halts the analysis. In this case, the initial backup and synchronization must be redone, which is both labor-intensive and increases the risk of data loss. To address this problem, the present invention provides a method for determining breakpoints in log files.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a log file breakpoint determination method provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where breakpoints in log files required for data backup and restart need to be determined. This method can be executed by a log file breakpoint determination device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0032] S110. When the backup service is detected to be running, obtain the pre-recorded current log breakpoint, read transaction data from the current log breakpoint of the log file and perform data backup operation.

[0033] As we know, databases are used to store critical data and provide data access services for upper-layer applications. To ensure data security, data from the source database is backed up in real-time to a backup database. If the source database becomes unavailable or loses data due to a failure, the data from the source database can be retrieved from the backup database. The process of backing up data from the source database to the backup database is called a backup service.

[0034] Considering that upper-layer applications' read and write accesses to the database will ultimately be transformed into multiple transaction data entries and recorded in log files, the backup service needs to synchronize this transaction data after it is started to ensure that the backup database always maintains consistency with the source database. Therefore, it needs to be able to completely read all transaction data generated by the source database since the initial synchronization point, correctly parse and reverse-engineer it into database operation statements, and re-execute them in the backup database. No transaction data can be missed during this process; otherwise, inconsistency between the backup and source databases will occur. In practice, the backup service may be paused, exited, or restarted. When the backup service restarts, it needs to start analyzing the incomplete transactions from before the last exit and re-analyze them to ensure no transaction data is missed. Therefore, during the backup service process, the starting position information of the transactions being analyzed needs to be saved to a disk file for easy re-analysis after the backup service is interrupted.

[0035] Taking Oracle database as an example, the Oracle redo log file records all transaction data generated by Oracle, ensuring the recovery of the Oracle database from failures. Each Oracle instance has a set of online redo log files that are used cyclically, and used redo log files can be archived. Each redo log file has a unique sequence number. Due to disk capacity limitations, earlier redo logs may have been deleted.

[0036] The current log breakpoint is used to record the position of a transaction in the log file. This information allows for precise location within the log file. The current log breakpoint can be used as the breakpoint address for transaction analysis by the backup service, allowing analysis to continue from that address. The current log breakpoint recorded in the disk file is typically the starting address of a transaction within the log file. When the backup service restarts and needs to resume analysis from the last point, it needs to know this current log breakpoint. In this embodiment, the starting position of incomplete backup transactions (excluding invalid transactions) is used as the current log breakpoint and stored in the disk file. When the backup service restarts, it can first obtain the current log breakpoint, read transaction data from that breakpoint, and perform data backup operations.

[0037] Preferably, the information of the current log breakpoint may include: the log file sequence number, the block number within the log file, and the offset within the block of the log file. The log file sequence number is a unique identifier for the log file, and the block number within the log file corresponds to the block number described in the database product. In this embodiment, the log breakpoint can be determined during the analysis of transaction data after the backup service is started. The timing for determining the current log breakpoint can be manually set; for example, it can be done when a transaction end marker is encountered, or when a transaction start marker is encountered. No specific limitation is made here.

[0038] Specifically, upon detecting that the backup service has started, the system retrieves the pre-recorded current log breakpoint from the disk, reads transaction data from the current log breakpoint in the log file, and performs a data backup operation. For example, using an Oracle database, a single Oracle database transaction may include a batch of Data Manipulation Language (DML) or Data Definition Language (DDL) statements. A transaction typically begins with a Begin operation and ends with a Commit or Rollback operation.

[0039] S120. During the backup operation, update the pre-created list of incomplete backup transactions and the list of invalid transactions.

[0040] The incomplete backup transaction list records one or more incomplete backup transactions. Each incomplete backup transaction contains information such as transaction number, transaction start position, and transaction start time. Additionally, if an incomplete backup transaction involves data operations during the backup process, this information is also recorded in the incomplete backup transaction list. The incomplete backup transaction list can be stored in memory and retrieved from memory upon backup service restart. The invalid transaction list records invalid transactions. Each invalid transaction contains information such as transaction number. The invalid transaction list can be stored in a disk file and retrieved from the disk file upon backup service restart.

[0041] In this embodiment, a transaction is defined as a long transaction if it spans more than the number of online log file groups from its start to its end. For example, a database may have 5 online file groups, which it uses cyclically to record log content. A transaction might begin with data in the first file group. As data is written to the log files, the database continues writing to the second file group after the first is full, and so on. If the transaction is still ongoing when the Nth file group is full, it is defined as a long transaction. The value of N can be customized; for example, N can be set to 3, meaning that if a transaction spans 3 log file groups from its start to its end, it is considered a long transaction. An incomplete long transaction that has not yet captured valid transaction data from its start to the current time is considered an invalid transaction. However, once valid transaction data is captured, the long transaction is no longer considered invalid.

[0042] Specifically, the process involves retrieving a list of incomplete backup transactions from memory and a list of invalid transactions from the disk file. After reading transaction data from the current log breakpoint in step S110 and performing data backup, the transaction data is parsed. When a new incomplete backup transaction is detected, it is updated to the list of incomplete backup transactions. When an incomplete transaction is found to have ended, the completed transaction is removed from the list. For example, during the parsing of transaction data, if a start message (such as begin) is detected, the transaction corresponding to that start message is considered a new transaction, and its transaction number, start position, and start time are recorded in the list of incomplete backup transactions. Similarly, if an end message (such as Commit or Rollback) is detected during the parsing of transaction data, the transaction corresponding to that end message is considered complete, and it should be removed from the list of incomplete backup transactions.

[0043] Furthermore, during the data backup operation, when a new invalid transaction is detected, it is updated to the invalid transaction list; when the operation data corresponding to an invalid transaction is detected, the invalid transaction is removed from the invalid transaction list. For example, if a transaction meets the invalid transaction criteria during transaction data parsing, it is considered a newly added invalid transaction and recorded in the invalid transaction list; if operation data of an invalid transaction in the invalid transaction list is detected during transaction data parsing, the transaction is considered not invalid and is removed from the invalid transaction list.

[0044] S130. When the breakpoint determination conditions are met, determine and save the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions.

[0045] The breakpoint determination conditions can be manually set. They can be determined upon encountering a certain type of message, or at set time intervals. Preferably, the breakpoint can be determined when an end message is encountered. For example, using an Oracle database, the breakpoint is determined when a commit or rollback operation message is encountered.

[0046] In existing technologies, the current log breakpoint is typically generated when the backup service encounters an end message. It checks the list of all incomplete transactions in memory, finds the earliest incomplete transaction, and sets its starting position as the current log breakpoint, recording it on disk. However, in reality, the log file pointed to by this current log breakpoint might have been generated a long time ago, and database administrators may periodically delete early log files to save disk space. This results in the backup service being unable to find the corresponding log file after restarting, preventing further analysis and causing the backup service to terminate. To continue backup, a new initial backup must be performed. To address this issue, this embodiment proposes adding an invalid transaction list to record all currently invalid transactions. From a data backup perspective, invalid transactions are considered transactions that can be ignored by the backup service, as ignoring them will not cause inconsistencies between the source and backup databases. Therefore, invalid transactions are filtered when generating the current breakpoint data, focusing only on valid and necessary transactions.

[0047] It is known that if such invalid transactions are not excluded, the next log breakpoint will be recorded much earlier, possibly recording a log file position from days or even weeks ago. The earlier the breakpoint is recorded, the greater the risk of the log file not being found. In this embodiment, by excluding invalid transactions based on the invalid transaction list, the recorded log breakpoint is often very close to the current time, greatly reducing the risk of the log file not being found. Skipping the analysis of those incomplete invalid long transactions will not lead to the loss of application data.

[0048] Specifically, the steps for determining the current log breakpoint in this implementation can be as follows: when the breakpoint determination conditions are met, based on invalid transactions among invalid transactions, after excluding invalid transactions in the list of incomplete backup transactions, the remaining transactions in the list of incomplete backup transactions are valid transactions that need attention. The transaction with the earliest start time is retrieved from the remaining transactions, and the starting position corresponding to that transaction is taken as the next log breakpoint and stored in a disk file. If the backup service restarts, the current log breakpoint can be obtained from the disk file to continue the analysis of transaction data.

[0049] It's important to note that when the breakpoint determination conditions are met, the next log breakpoint is determined and saved to the disk file. Each time the breakpoint determination conditions are met, a log breakpoint is determined, and the determined log data is saved to the disk file, overwriting the previously determined log breakpoint. In other words, the disk file always stores the most recent log breakpoint. If the backup service is running normally, the log breakpoints saved in the disk file will be continuously updated as transaction data is analyzed.

[0050] This invention discloses a method for determining breakpoints in log files. The method includes: upon detecting the start of a backup service, obtaining a pre-recorded current log breakpoint; reading transaction data from the current log breakpoint in the log file and performing a data backup operation; during the backup operation, updating a pre-created list of incomplete backup transactions and a list of invalid transactions; and when breakpoint determination conditions are met, determining and saving the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions. Considering that in existing technologies, after a backup service restart, the log file corresponding to the breakpoint may have been cleaned up and cannot be backed up normally, this solution addresses this problem by excluding many long-term incomplete and invalid transactions based on the invalid transaction list. By ignoring invalid long transactions when determining breakpoints, the recorded log breakpoints are usually very close to the current time, greatly reducing the risk of not finding the log file. This avoids the backup service restarting and ensures normal backup after restarting.

[0051] Example 2

[0052] Figure 2 This is a flowchart of a log file breakpoint determination method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment. In this embodiment, updating the pre-created backup incomplete transaction list and invalid transaction list during the backup operation execution process can be specifically described as follows: During the backup operation execution process, newly determined backup incomplete transactions are added to the pre-created backup incomplete transaction list, and backup completed transactions are deleted from the backup incomplete transaction list; During the backup operation execution process, invalid transactions determined from the log file are added to the pre-created invalid transaction list, and the determined target transactions are deleted from the invalid transaction list; The target transaction is an invalid transaction whose transaction label changes from invalid to valid.

[0053] In addition, this embodiment can also determine and save the next log breakpoint corresponding to the next startup of the backup service when the breakpoint determination conditions are met, based on the list of incomplete backup transactions and the list of invalid transactions. Specifically, the transactions included in the list of incomplete backup transactions but not included in the list of invalid long transactions constitute a candidate transaction set; based on the number of transactions included in the candidate transaction set, the next log breakpoint corresponding to the next startup of the backup service is determined.

[0054] like Figure 2 As shown in Embodiment 2 of the present invention, a method for determining breakpoints in a log file specifically includes the following steps:

[0055] S210. When the backup service is detected to be running, obtain the pre-recorded current log breakpoint, read transaction data from the current log breakpoint of the log file and perform data backup operation.

[0056] S220. During the backup operation, add newly determined backup incomplete transactions to the pre-created backup incomplete transaction list, and delete backup completed transactions from the backup incomplete transaction list.

[0057] In this embodiment, during the backup operation, the transaction data recorded in the log file is read and parsed sequentially. Since a transaction generally includes a start message, an operation message, and an end message, when a start message is encountered during parsing, it indicates a new backup incomplete transaction, which needs to be added to the backup incomplete transaction list. When an end message is encountered during parsing, it indicates a completed transaction, which is a backup completed transaction, and this transaction needs to be deleted from the backup incomplete transaction list.

[0058] S230. During the backup operation, add invalid transactions identified from the log file to the pre-created invalid transaction list, and delete the identified target transactions from the invalid transaction list.

[0059] The target transaction is the invalid transaction whose transaction label has been changed from invalid to valid. A transaction label can be understood as a label that indicates whether a transaction is valid.

[0060] In this step, during the backup operation, all transactions in the incomplete backup transaction list are checked. If a transaction is found that meets the conditions for a long transaction, and the amount of data operated on by this transaction is zero, and it has not yet been recorded in the invalid transaction list, then this transaction is considered a newly added invalid transaction, and is added to the invalid transaction list. Preferably, for operational efficiency, the refresh of the invalid transaction list can be placed during the processing of end messages. Whenever a transaction end message is encountered, all currently incomplete backup transactions are checked for invalid transactions. If a valid transaction is found and has not yet been recorded in the invalid transaction list, its transaction number is added to the invalid transaction list.

[0061] It is important to know that during the backup operation, if a transaction is in the invalid transaction list, and a transaction message for the corresponding operation data is encountered, then the transaction is considered a valid transaction. The transaction label of the transaction changes from invalid to valid, and the transaction needs to be removed from the invalid transaction list.

[0062] S240. Transactions that are included in the list of incomplete backup transactions but not in the list of invalid transactions constitute a candidate transaction set.

[0063] In this embodiment, the backup incomplete transaction list records all currently incomplete transactions, and the invalid transaction list contains invalid transactions. When determining the log breakpoint, invalid transactions are first excluded, and the remaining transactions in the backup incomplete transaction list after excluding invalid transactions are determined as valid transactions that need attention to form a candidate transaction set.

[0064] The transactions in the candidate transaction set are determined jointly by the list of incomplete backup transactions and the list of invalid transactions. The final candidate transaction set may contain one or more transactions, or it may be empty. For example, assuming the list of incomplete backup transactions contains transactions 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10, and the list of invalid transactions contains transactions 1, 3, 12, and 13, the transactions included in the list of incomplete backup transactions but not in the list of invalid transactions constitute the candidate transaction set, which includes transactions 2, 4, 5, 6, 7, 8, 9, and 10. Alternatively, assuming the list of incomplete backup transactions contains transactions 20 and 21, and the list of invalid transactions contains transactions 20, 21, 30, and 31, the transactions included in the list of incomplete backup transactions but not in the list of invalid transactions constitute the candidate transaction set, which would be empty.

[0065] S250. Based on the number of transactions contained in the candidate transaction set, determine the next log breakpoint corresponding to the next startup of the backup service.

[0066] It's clear that the candidate transaction set may or may not contain transactions. If it does contain transactions, the next log breakpoint for the backup service's next startup can be determined based on the information about those transactions. For example, the start times of all transactions in the candidate transaction set can be compared, and the starting position of the transaction with the earliest start time can be used as the next log breakpoint. The purpose of this setting is to ensure that all incomplete transactions in the candidate transaction set are parsed, thus preventing transaction data from being missed, by using the starting position of the transaction with the earliest start time as the next log breakpoint and continuing parsing from this breakpoint.

[0067] As an optional embodiment of this application, based on the above embodiments, the steps for determining invalid transactions from the log file may include:

[0068] S221. When it is detected that the information currently read from the log file is transaction start information, the transaction corresponding to the transaction start information is determined as the current backup transaction.

[0069] Specifically, transaction start information can be understood as information indicating the beginning of a transaction. For example, in Oracle databases, a transaction begins with a "Begin" operation. When reading and parsing transaction data in the log file, if the information currently read from the log file is detected to be transaction start information, then the transaction corresponding to the transaction start information is identified as the current backup transaction.

[0070] S222. If no valid transaction data is detected after reading the data of the current backup transaction for the set data length, the current backup transaction is determined to be invalid and the corresponding transaction tag is marked as invalid.

[0071] The set data length can be understood as the length corresponding to an invalid transaction. Valid transaction data can be understood as operation data, such as DML or DDL operation data in an Oracle database transaction. If the set data length is read from the current backup transaction, it can be determined that the invalid transaction is a long transaction. If no corresponding valid transaction data is detected after reading the set data length from the current backup transaction, then the current backup transaction is determined to be an invalid transaction, and the corresponding transaction tag is marked as invalid. Therefore, once an invalid transaction is identified, it needs to be updated to the invalid transaction list.

[0072] As an optional embodiment of this application, based on the above embodiments, the step of determining the target transaction may include:

[0073] For each invalid transaction in the invalid transaction list, if valid transaction data for the invalid transaction is read from the log file, the invalid transaction is marked as the target transaction, and the corresponding transaction label is changed from invalid to valid.

[0074] In this step, during the backup service, for each invalid transaction in the invalid transaction list, if valid transaction data corresponding to an invalid transaction is read from the log file, it indicates that the invalid transaction corresponds to valid transaction data. Therefore, the invalid transaction is recorded as the target transaction, and the corresponding transaction label is changed from invalid to valid. It can be seen that once the target transaction is identified, it needs to be removed from the invalid transaction list.

[0075] As an optional embodiment of this application, based on the above embodiments, the next log breakpoint corresponding to the next startup of the backup service is determined based on the number of transactions contained in the candidate transaction set, including:

[0076] S241. If the candidate transaction set is empty, then the current read point of the log file is determined as the next log breakpoint.

[0077] Considering that if the candidate transaction set is empty, it means that there is no transaction that is neither an invalid transaction nor a backup incomplete transaction, that is, there is no valid backup incomplete transaction, then the current read point of the log file is determined as the next log breakpoint.

[0078] S242. If the candidate transaction set is not empty, then determine the next log breakpoint based on the transaction start data corresponding to each transaction in the candidate transaction set.

[0079] The transaction start data includes: transaction number, transaction start time, and transaction start position. If the candidate transaction set is not empty, the next log breakpoint can be determined based on the transaction start data corresponding to each transaction in the candidate transaction set. To ensure that no transaction data is missed after the backup service restarts, it is necessary to ensure that reading from the determined next log breakpoint ensures that all transactions in the candidate transaction set are read. Therefore, the next log breakpoint is determined based on the transaction start time and transaction start position corresponding to each transaction in the candidate transaction set.

[0080] Optionally, if the candidate transaction set is not empty, the next log breakpoint is determined based on the transaction start information corresponding to each transaction in the candidate transaction set, including: determining the transaction corresponding to the earliest transaction start time from each transaction as the first transaction; and taking the start position of the first transaction as the next log breakpoint.

[0081] Specifically, in order to ensure that all transactions in the candidate transaction set can be read and parsed after the backup service restarts, the transaction with the earliest start time is determined as the first transaction, and the start position of the first transaction is used as the next log breakpoint.

[0082] An example description is given using a specific application scenario. Figure 3 This is a flowchart illustrating the steps involved in restarting an Oracle database backup service. For example, using the Oracle database backup service, an Oracle database transaction may consist of a series of DML or DDL operations. A transaction typically begins with a Begin operation and ends with a Commit or Rollback operation, and its data is recorded in the redo log file. Figure 3 As shown, the specific steps after restarting the Oracle database backup service are as follows:

[0083] Step a) Once the backup service is detected to be running, obtain the pre-recorded current log breakpoint;

[0084] Step b) Read transaction data from the current log breakpoint in the log file and perform a data backup operation;

[0085] Step c) Determine whether the transaction data being read is Begin, DML / DDL, or Commit / Rollback;

[0086] Step d) If the transaction data is Begin, then add the newly determined backup incomplete transaction to the list of pre-created backup incomplete transactions.

[0087] Step e) If the transaction data is DML / DDL, then determine whether the current transaction exists in the invalid transaction list;

[0088] Step f) If it exists, remove the current transaction from the list of invalid transactions;

[0089] Step g) If the transaction data is Commit / Rollback, then delete the completed backup transaction from the list of incomplete backup transactions;

[0090] Step h) Retrieve the list of currently incomplete transactions. If a new invalid long transaction is found, add the transaction to the list of invalid transactions.

[0091] Step i) Retrieve the list of incomplete backup transactions and exclude transactions that exist in the list of invalid transactions to obtain the earliest valid transaction (assuming that the number of valid transactions is not zero).

[0092] Step j) Set the starting position of the earliest valid transaction as the next log breakpoint.

[0093] As an optional embodiment of the present invention, this embodiment specifies the process of updating a pre-created list of incomplete backup transactions and a list of invalid transactions during the backup operation, and the process of determining the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions when the breakpoint determination conditions are met. This solution excludes many long-term incomplete and invalid transactions through the invalid transaction list, so the recorded log breakpoint will not be the starting point of these long-term invalid transactions. If such long-term invalid transactions are not excluded, the recorded transaction analysis breakpoint will be much earlier, possibly recording the location of the log file from several days or even weeks ago. The earlier the recorded analysis breakpoint, the greater the risk of the log file not being found. After excluding invalid transactions through the invalid transaction list, the recorded analysis breakpoint is often very close to the current time, greatly reducing the risk of the log file not being found. Skipping the analysis of those incomplete, invalid, long transactions will not lead to the loss of application data. This solution adds an invalid transaction list, which is simple and easy to implement.

[0094] Example 3

[0095] Figure 4 This is a schematic diagram of a log file breakpoint determination device provided in Embodiment 3 of the present invention. This device is suitable for determining breakpoints in log files required during data backup and restart. The device can be implemented by software and / or hardware and is generally integrated into an electronic device. Figure 4 As shown, the device includes: a data backup operation module 31, a transaction list update module 32, and a breakpoint determination module 33.

[0096] Among them, the data backup operation module 31 is used to obtain the pre-recorded current log breakpoint after the backup service is detected to start, read transaction data from the current log breakpoint of the log file and perform data backup operation;

[0097] The transaction list update module 32 is used to update the pre-created list of incomplete backup transactions and the list of invalid transactions during the backup operation.

[0098] The breakpoint determination module 33 is used to determine and save the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions when the breakpoint determination conditions are met.

[0099] Optionally, the transaction list update module 32 may include:

[0100] The backup incomplete transaction list update unit is used to add newly determined backup incomplete transactions to the pre-created backup incomplete transaction list and delete backup completed transactions from the backup incomplete transaction list during the backup operation execution process;

[0101] The invalid transaction list update unit is used to add invalid transactions identified from the log file to a pre-created invalid transaction list during the backup operation process, and to delete the identified target transactions from the invalid transaction list.

[0102] The target transaction is an invalid transaction whose transaction label has been changed from invalid to valid.

[0103] Furthermore, the backup incomplete transaction list update unit can be specifically used for:

[0104] When it is detected that the information currently read from the log file is transaction start information, the transaction corresponding to the transaction start information is determined as the current backup transaction;

[0105] If no valid transaction data is detected after reading the data of the current backup transaction for the set length, the current backup transaction is determined to be invalid, and the corresponding transaction tag is marked as invalid.

[0106] Furthermore, the invalid transaction list update unit can specifically be used for:

[0107] For each invalid transaction in the invalid transaction list, if valid transaction data for the invalid transaction is read from the log file, the invalid transaction is marked as the target transaction, and the corresponding transaction label is changed from invalid to valid.

[0108] Optionally, the breakpoint determination module 33 may include:

[0109] The candidate transaction set determination unit is used to form a candidate transaction set from transactions that are included in the backup incomplete transaction list and are not included in the invalid long transaction list;

[0110] The next log breakpoint determination unit is used to determine the next log breakpoint corresponding to the next startup of the backup service based on the number of transactions contained in the candidate transaction set.

[0111] Furthermore, the unit for determining the candidate transaction set can specifically be used for:

[0112] If the candidate transaction set is empty, the current read point of the log file will be determined as the next log breakpoint;

[0113] If the candidate transaction set is not empty, the next log breakpoint is determined based on the transaction start information corresponding to each transaction in the candidate transaction set.

[0114] Furthermore, the next log breakpoint determination unit can specifically be used for:

[0115] The transaction with the earliest start time among all transactions is identified as the first transaction.

[0116] Set the starting position of the first transaction as the next log breakpoint.

[0117] The log file breakpoint determination device provided in this embodiment of the invention can execute the log file breakpoint determination method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] Example 4

[0119] Figure 5 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0120] like Figure 5 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0121] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0122] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as the breakpoint determination method for log files.

[0123] In some embodiments, the log file breakpoint determination method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the log file breakpoint determination method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to execute the log file breakpoint determination method by any other suitable means (e.g., by means of firmware).

[0124] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0126] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0127] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0128] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0129] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0130] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for determining breakpoints in a log file, characterized in that, include: When the backup service is detected to be started, the pre-recorded current log breakpoint is obtained, transaction data is read from the current log breakpoint in the log file, and data backup operation is performed. During the backup operation, update the pre-created list of incomplete backup transactions and the list of invalid transactions; When the breakpoint determination conditions are met, the next log breakpoint corresponding to the next startup of the backup service is determined and saved based on the list of incomplete backup transactions and the list of invalid transactions.

2. The method according to claim 1, characterized in that, The step of updating the pre-created list of incomplete backup transactions and the list of invalid transactions during the backup operation includes: During the backup operation, newly determined incomplete backup transactions are added to the pre-created list of incomplete backup transactions, and completed backup transactions are deleted from the list of incomplete backup transactions. During the backup operation, invalid transactions identified from the log file are added to a pre-created list of invalid transactions, and the identified target transactions are removed from the list of invalid transactions. The target transaction is an invalid transaction whose transaction label has been changed from invalid to valid.

3. The method according to claim 2, characterized in that, The steps for identifying invalid transactions from the log file include: When it is detected that the information currently read from the log file is transaction start information, the transaction corresponding to the transaction start information is determined as the current backup transaction; If no valid transaction data is detected after reading the current backup transaction for the set data length, the current backup transaction is determined to be invalid, and the corresponding transaction tag is marked as invalid.

4. The method according to claim 2, characterized in that, The steps for determining the target transaction include: For each invalid transaction in the invalid transaction list, if valid transaction data of the invalid transaction is read from the log file, the invalid transaction is recorded as the target transaction, and the corresponding transaction label is changed from invalid to valid.

5. The method according to claim 1, characterized in that, When the breakpoint determination conditions are met, the next log breakpoint corresponding to the next startup of the backup service is determined and saved based on the list of incomplete backup transactions and the list of invalid transactions, including: Transactions that are included in the list of incomplete backup transactions but not included in the list of invalid transactions constitute a candidate transaction set; Based on the number of transactions contained in the candidate transaction set, determine the next log breakpoint corresponding to the next startup of the backup service.

6. The method according to claim 5, characterized in that, The step of determining the next log breakpoint corresponding to the next startup of the backup service based on the number of transactions contained in the candidate transaction set includes: If the candidate transaction set is empty, the current read point of the log file will be determined as the next log breakpoint; If the candidate transaction set is not empty, then the next log breakpoint is determined based on the transaction start data corresponding to each transaction in the candidate transaction set.

7. The method according to claim 6, characterized in that, If the candidate transaction set is not empty, then based on the transaction start data corresponding to each transaction in the candidate transaction set, the next log breakpoint is determined, including: The transaction with the earliest start time among all the transactions is identified as the first transaction. Set the starting position of the first transaction as the next log breakpoint.

8. A breakpoint determination device for log files, characterized in that, include: The data backup operation module is used to obtain the pre-recorded current log breakpoint after the backup service is detected to be started, read transaction data from the current log breakpoint in the log file and perform data backup operation; The transaction list update module is used to update the pre-created list of incomplete backup transactions and the list of invalid transactions during the backup operation. The breakpoint determination module is used to determine and save the next log breakpoint corresponding to the next startup of the backup service based on the list of incomplete backup transactions and the list of invalid transactions when the breakpoint determination conditions are met.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the log file breakpoint determination method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the breakpoint determination method for the log file as described in any one of claims 1-7.

Images