Method, apparatus, and medium for database log processing

Abstract

The present disclosure relates to a database log processing method, device, electronic equipment and medium, and relates to distributed storage and cloud computing technology, the method comprises: obtaining an attribute value of a delete lock attribute of an operation log in a database; the delete lock attribute is used to describe a backup state of the operation log in the database; according to the attribute value, determining an incremental operation log of the database; performing backup processing on the incremental operation log; if the backup of the incremental operation log is successful, updating the attribute value of the delete lock attribute; according to the attribute value of the delete lock attribute, recycling the operation log storage space. This method can ensure that part of the operation log recycled during storage is already backed up, regardless of how frequently the operation log is updated, and can achieve real-time and complete backup of the operation log of the database, thereby meeting the requirement of data rollback to any time point.

Description

Technical Field

[0001] This disclosure relates to the fields of database, distributed storage and cloud computing technologies, and in particular to a method, apparatus, electronic device and medium for database log processing. Background Technology

[0002] As data storage scales increase, data security requirements become more stringent, and the need for timely responses in high-concurrency data query scenarios grows, the performance requirements for data storage services also increase. Distributed storage architectures help improve data storage capacity, disaster recovery, high availability, and flexible scalability. Databases such as MySQL, Oracle, Redis, and MongoDB are based on distributed storage technologies and mostly adopt master-slave architectures, sharded cluster architectures, and replica set architectures to implement data storage.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following technical problems in the related technologies: In the related technologies, users have a need to roll back data to a certain point in history, which requires the database to support data rollback to any point in time; In the limited storage space corresponding to the operation log (e.g., represented by oplog in MongoDB database), new data usually overwrites some historical data, that is, the storage space of some historical data is reclaimed to store new data. However, as data operations (such as adding, deleting, querying, or modifying) in the database are relatively frequent, this may cause some historical data to be deleted before it has been backed up in order to release storage space and store constantly updated new data. Thus, due to the lack of operation logs, the database cannot support data rollback to any point in time. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, embodiments of this disclosure provide a method, apparatus, electronic device, and medium for database log processing.

[0005] In a first aspect, embodiments of this disclosure provide a method for processing database logs. The method includes: obtaining the attribute value of a delete lock attribute of an operation log in the database; the delete lock attribute being used to describe the backup status of the operation log in the database; determining an incremental operation log of the database based on the attribute value; performing backup processing on the incremental operation log; updating the attribute value of the delete lock attribute if the incremental operation log backup is successful; and reclaiming the operation log storage space based on the attribute value of the delete lock attribute.

[0006] According to embodiments of this disclosure, the backup status includes: time information of the backed-up operation log; the attribute value of the delete lock attribute includes: the latest timestamp of the backed-up operation log; or, the attribute value of the delete lock attribute includes: the coverage period information of the backed-up operation log. The delete lock attribute value is dynamically updated as incremental operation logs are backed up. Before and during the incremental operation log backup, the attribute value uses the result before the update; after the incremental operation log backup is successful, the attribute value uses the result after the update.

[0007] According to embodiments of this disclosure, when the attribute value includes the latest timestamp of the backed-up operation log, the operation log storage space is reclaimed based on the attribute value of the deletion lock attribute, including: for the target operation log storage space to be reclaimed, obtaining the timestamp of the operation log in the target operation log storage space; determining the temporal relationship between the latest time value of the timestamp and the latest timestamp; deleting the operation log in the target operation log storage space when the temporal relationship indicates that the latest time value is earlier than or equal to the latest timestamp; and retaining the target operation log storage space when the temporal relationship indicates that the latest time value is later than the latest timestamp.

[0008] According to embodiments of this disclosure, when the attribute value includes the coverage period information of the backed-up operation log, the operation log storage space is reclaimed based on the attribute value of the deletion lock attribute, including: for the target operation log storage space to be reclaimed, obtaining the time period of the timestamp of the operation log in the target operation log storage space; determining the interval relationship between the time period of the timestamp and the coverage period of the coverage period information; when the interval relationship indicates that the time period of the timestamp is a subset or the entirety of the coverage period, deleting the operation log in the target operation log storage space; when the interval relationship indicates that there is an overlap between the time period of the timestamp and the coverage period, deleting the operation log corresponding to the overlap period and retaining the operation log in the remaining time periods; when the interval relationship indicates that there is no overlap between the time period of the timestamp and the coverage period, retaining the target operation log storage space.

[0009] According to embodiments of this disclosure, determining the incremental operation logs of the database based on the aforementioned attribute values ​​includes: filtering first operation logs in the database whose log timestamps are after the latest timestamp sequence; and selecting part or all of the first operation logs as the incremental operation logs. Alternatively, determining the incremental operation logs of the database based on the aforementioned attribute values ​​includes: filtering second operation logs in the database whose log timestamps are outside the aforementioned coverage period information; and selecting part or all of the second operation logs as the incremental operation logs.

[0010] According to embodiments of this disclosure, when the incremental operation log backup is successful, updating the attribute value of the delete lock attribute includes: determining the latest time-series value in the timestamp of the incremental operation log as the attribute value update result, or determining the union of the newly added coverage period of the incremental operation log and the existing coverage period of the backed-up operation log as the attribute value update result; and updating the attribute value of the delete lock attribute to the attribute value update result.

[0011] According to embodiments of this disclosure, the method further includes: recording the update operation of updating the attribute value of the deleted lock attribute in the operation log of the database; wherein the database is the database corresponding to the master storage node in a storage cluster built using one of the following architectures: master-slave mode architecture, replica set architecture, sharded cluster architecture, and a combination of replica set and sharded cluster architecture; the operation log is used for synchronization between the master storage node and the slave storage node; and when the first slave storage node switches to the master storage node, the operation log is backed up and the operation log storage space is reclaimed for the database of the first slave storage node.

[0012] According to embodiments of this disclosure, the method further includes: receiving an operation rollback request, the operation rollback request carrying location information indicating the rollback destination; querying historical operation logs corresponding to the location information in the operation log storage space; if no historical operation logs are found, querying historical operation logs corresponding to the location information from a backup operation log; determining the data status corresponding to the location information based on the historical operation logs; and generating an operation rollback result based on the data status.

[0013] Secondly, embodiments of this disclosure provide an apparatus for database log processing. The apparatus includes: an attribute value acquisition module, an incremental log determination module, a backup module, an attribute value update module, and a storage space reclamation module. The attribute value acquisition module acquires the attribute value of a delete lock attribute of an operation log in the database; the delete lock attribute describes the backup status of the operation log in the database. The incremental log determination module determines the incremental operation log of the database based on the attribute value. The backup module backs up the incremental operation log. The attribute value update module updates the attribute value of the delete lock attribute when the incremental operation log backup is successful. The storage space reclamation module reclaims the storage space of the operation log based on the attribute value of the delete lock attribute.

[0014] Thirdly, embodiments of this disclosure provide an electronic device. The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, communication interface, and memory communicate with each other via the communication bus; the memory stores computer programs; and the processor, when executing the program stored in the memory, implements the database log processing method described above.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the database log processing method described above.

[0016] The technical solutions provided in the embodiments of this disclosure have at least some or all of the following advantages:

[0017] By setting a delete lock attribute for the operation log, which describes the backup status of the operation log in the database, the backup status indicates which operation logs have been backed up completely. This clearly indicates that only operation logs that have been backed up can be deleted, thus realizing the reclamation of storage space. Based on the above delete lock attribute setting, a corresponding relationship is effectively established between the operation log, the latest backup status, and whether storage space can be reclaimed. Thus, as the operation log is continuously generated, based on incremental backup, the current delete lock attribute value is used to determine the incremental operation logs that need to be incrementally backed up on top of the already backed-up operation logs. After the incremental operation log is successfully backed up, the delete lock attribute value is updated to obtain the latest backup status. At the same time, the operation log storage space can be reclaimed based on the delete lock attribute at each moment. No matter how frequently the operation log is updated, it is always guaranteed that the operation logs to be reclaimed during the storage process are those that have already been backed up, realizing real-time and complete backup of the database operation log, thereby meeting the need for data rollback to any point in time. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 The system architecture of the database log processing method and apparatus applicable to embodiments of this disclosure is schematically illustrated;

[0021] Figure 2 A flowchart illustrating a method for processing database logs according to an embodiment of the present disclosure is shown schematically.

[0022] Figure 3A A detailed implementation flowchart of step S220 according to an embodiment of the present disclosure is illustrated schematically;

[0023] Figure 3B A detailed implementation flowchart of step S220 according to another embodiment of the present disclosure is illustrated schematically;

[0024] Figure 4A A detailed implementation flowchart of step S250 according to an embodiment of the present disclosure is shown schematically;

[0025] Figure 4B A detailed implementation flowchart of step S250 according to another embodiment of the present disclosure is illustrated schematically;

[0026] Figure 5 A flowchart illustrating a method for database log processing according to another embodiment of this disclosure is shown schematically;

[0027] Figure 6 A schematic block diagram of a database log processing apparatus according to an embodiment of the present disclosure is shown; and

[0028] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0030] Figure 1 The system architecture of the method and apparatus for database log processing applicable to embodiments of this disclosure is illustrated schematically.

[0031] Reference Figure 1 As shown, the system architecture 100 of the database log processing method and apparatus applicable to embodiments of this disclosure includes: a service request end 110 and a service node 120.

[0032] The service demand side 110 can be any entity that needs data storage services, such as terminal devices or servers that require data storage services.

[0033] Service node 120 is used to provide data storage services to service requester 110. In some embodiments, refer to Figure 1 As shown, service node 120 may include: backup process 121 and primary storage node 122. Backup process 121 is used to perform incremental backups of the operation logs in the primary storage node. Primary storage node 122 can correspond to various types of databases. This primary storage node 122 can be a storage node in a distributed storage architecture, for example, in... Figure 1 The example shown is the master node in service cluster 130. Service cluster 130 can be a storage cluster built using one of the following architectures: master-slave mode, replica set, sharded cluster, or a combination of replica set and sharded cluster. Service cluster 130 also includes multiple slave nodes, such as slave storage node 131 and slave storage node 132. The database log processing method provided in the embodiments of this disclosure can be executed by the aforementioned service node 120. A corresponding initialization script is pre-written in service node 120, which can cover functions such as starting the master node and starting incremental backup tasks. Correspondingly, the apparatus for database log processing can be the aforementioned service node.

[0034] A master-slave architecture refers to a service cluster divided into two roles: a master node (which can also be understood as the master database) and slave nodes (which can also be understood as subordinate databases). Generally, the master node interacts with the service requesting end, supporting data reading and writing; the slave nodes synchronize data with the master node, essentially acting as a copy of the master node's data, typically in read-only mode. In a master-slave architecture, if the master node fails (e.g., crashes), external means are needed to handle the situation or a new master node needs to be designated.

[0035] A replica set architecture has one master node and multiple slave nodes, similar to the master-slave model. However, the difference between a replica set and the master-slave model is that when the master node in the cluster fails, the nodes within the replica set can automatically vote to elect a new master node and guide the remaining slave nodes to connect to the new master node.

[0036] The sharded cluster architecture consists of: routing nodes (MongoS, which can be understood as routing servers), configuration shards (Config, which can be understood as configuration servers), and replica set shards (which can be understood as servers that provide data storage services). The replica set is the basic unit.

[0037] In a sharded cluster architecture, a replica set shard comprises a subset of data to be stored. This means that the large-scale data to be stored is distributed into different subsets, each stored in a different shard. Each shard can be deployed as a replica set; if each shard is deployed as a replica set, it forms a combined replica set and sharded cluster architecture. MongoS acts as a query router, providing an interface between client applications and the sharded cluster. The shard configuration server acts as a configuration server, storing cluster metadata and configuration settings.

[0038] The method provided in this disclosure can be applied to various types of databases, such as MongoDB databases, which can adopt a replica set architecture, a sharded cluster architecture, or a combination of replica set and sharded cluster architectures. In some embodiments, it can also be applied to a master-slave architecture.

[0039] In some embodiments, refer to Figure 1 As shown, the various processes of a MongoDB replica set or sharded cluster are deployed based on Docker (a container orchestration tool). For example, the replica processes of MongoS, MongoD, and Config are deployed in independent containers.

[0040] The aforementioned service node 120 includes not only data storage components, but also, in addition to the replica process corresponding to the primary storage node 122, management scripts and processes such as backup process 121, to implement data storage and backup functions. This embodiment uses the example of backup process 121 and primary storage node 122 as two independent parts. In other embodiments, backup modules can be set up within the primary storage node according to actual needs. The system architecture disclosed herein is not limited to the system architecture 100 of the above example, but is merely described as an exemplary system architecture.

[0041] The first exemplary embodiment of this disclosure provides a method for database log processing. This method can be executed by a service node 120 in system architecture 100.

[0042] Figure 2 A flowchart illustrating a method for processing database logs according to an embodiment of the present disclosure is shown schematically.

[0043] Reference Figure 2 As shown, the database log processing method provided in this embodiment includes the following steps: S210, S220, S230, S240 and S250.

[0044] In step S210, the attribute value of the delete lock attribute of the operation log in the database is obtained; the delete lock attribute is used to describe the backup status of the operation log in the database.

[0045] In some embodiments, operations such as adding, deleting, and modifying data in the database generate operation information, which is recorded in the operation log. The operation log has pre-allocated storage space. In some embodiments, query operations are not recorded in the operation log.

[0046] For example, in a database, the oplog object is used as the data structure for storing operation logs, and the contents of each operation record are stored under this oplog object.

[0047] By adding the attribute field `oplogDeleteGuard` (an identifier name for an example of the delete lock attribute) to the `oplog` object, this delete lock attribute field serves as a global identifier variable for the `oplog`, used to describe the backup status of the operation log (e.g., which data has been backed up, which can be determined by the latest timestamp of the backed-up operation log content), and is used as a criterion for determining whether the data can be deleted during the `oplog`'s storage space reclamation process.

[0048] As an example of operation log content, users P1 and P2 both have edit permissions for data in a shared document W, which is stored in database K. At time t0 (e.g., 11:50), user P1 first adds the value R1 to field R in shared document W; at time t1 (e.g., 12:00), user P1 first writes the value M1 to field A in shared document W; then at time t2 (t2 is later than t1, e.g., 12:01), user P1 modifies the value of field A to M2; then at time t3 (e.g., 12:05), user P1 first writes the value N1 to field B in shared document W; then at time t4 (e.g., 12:08), user P2 modifies the value of field A to M3; finally, at time t5, user P2 adds a verification tag to the value N1 of field B.

[0049] According to the preset logic, all operation information on data in the database will be synchronously recorded in the database operation log. The operation information may include, but is not limited to, the following: operation user, operation time, data object operated on, execution action, and operation result. For example, the operation log content corresponding to the operation in the above example could be: {log1: User P1, time t1, field A, first write value, M1}, {log2: User P2, time t2, field A, modified value, changed M1 to M2}, {log3: User P3, time t3, field B, first write value, N1}, {log4: User P2, time t4, field A, modified value, changed M2 to M3}, {log5: User P2, time t5, field B, added tag, added a verified tag for field B}.

[0050] In some embodiments, the backup status includes the time information of the backed-up operation log, and the attribute value of the delete lock attribute includes the latest timestamp of the backed-up operation log. For example, the attribute value of the delete lock attribute can be represented by a timestamp to identify the oplog time point that has been backed up, and can also indicate that oplog data before this timestamp can be deleted. In this embodiment, the operation log is backed up sequentially according to time sequence, so only the latest timestamp of the backed-up operation log needs to be stored. This can effectively serve as the basis for reclamation of operation log storage space and the time point for further backup, and can also save the storage space of the operation log attribute fields and reduce the representation complexity of the attribute. It effectively simplifies the comparison logic when determining space reclamation and deletion, and has high processing efficiency.

[0051] In other embodiments, the attribute value of the deletion lock may further include: the overwrite period information of the backed-up operation log. In this embodiment, there is no need to restrict the operation log to be backed up in chronological order; skip backups can be performed according to the actual situation of storage space, that is, the time period of the xth backup of the operation log and the time period of the x+1th backup of the operation log may be discontinuous.

[0052] In the embodiments of this disclosure, the attribute value of the delete lock is dynamically updated as the incremental operation log is backed up. Before and during the backup of the incremental operation log, the attribute value adopts the result before the update; after the backup of the incremental operation log is successful, the attribute value adopts the result after the update.

[0053] For example, as an example of deleting lock attributes, the operation log storage space is 5G in total. Among them, the storage space of 0~1G stores the operation information set LOG0 within the time period T0~T1 (for example, a left-open right-closed interval, the right endpoint value has operation records, and the description of subsequent time periods follows the left-open right-closed method), the storage space of 1G~2G stores the operation information set LOG1 within the time period T1~T2, the storage space of 2G~3G stores the operation information set LOG2 within the time period T2~T3, and the storage space of 3G~4G stores the operation information set LOG3 within the time period T3~T4.

[0054] At time T4 (which is later than time T3, e.g., T4 = T3 + 50 seconds), assuming the data in LOG0 has been successfully backed up, the backup status of the operation log described by the delete lock attribute at time T4 will be: LOG0 has been backed up successfully. The attribute value of the delete lock attribute at time T4 can be represented as the latest timestamp of the operation log in the backed-up operation log set LOG0, for example, timestamp T1.

[0055] In step S220, the incremental operation log of the database is determined based on the aforementioned attribute values.

[0056] Figure 3A A detailed implementation flowchart of step S220 according to an embodiment of the present disclosure is shown schematically.

[0057] According to some embodiments of this disclosure, refer to Figure 3A As shown, when the attribute value of the deleted lock is the latest timestamp of the backed-up operation log, in step S220 above, the incremental operation log of the database is determined according to the attribute value, including the following steps: S311 and S312.

[0058] In step S311, the first operation log whose log timestamp is after the latest timestamp sequence is filtered in the database.

[0059] In step S312, select part or all of the first operation log as the incremental operation log.

[0060] When the delete lock's attribute value is the latest timestamp of the backed-up operation log, the value of the `oplogDeleteGuard` attribute of this delete lock can determine which time periods (all timestamps before the latest timestamp have been backed up) of the operation logs in the database have been backed up, and from which point in time the backup of new operation logs needs to begin. The aforementioned incremental operation logs refer to operation logs that require further backup based on the already backed-up operation logs; they are not necessarily operation logs generated in real-time incrementally. These incremental operation logs can be part or all of the data remaining in the unbacked-up operation logs after subtracting the backed-up operation logs from the total operation logs up to the current moment.

[0061] For example, at time T4, it can be determined that the incremental operation log is part or all of the operation log generated after time T1 (e.g., containing the operation log contents of LOG1, LOG2, LOG3 and time T4 (if any)). Here, LOG1 is used as an example of the incremental operation log.

[0062] Figure 3B A detailed implementation flowchart of step S220 according to another embodiment of the present disclosure is shown schematically.

[0063] According to other embodiments of this disclosure, refer to Figure 3B As shown, when the attribute value of the deleted lock is the overwrite period information of the backed-up operation log, in the above step S220, the incremental operation log of the above database is determined according to the above attribute value, including the following steps: S321 and S322.

[0064] In step S321, the second operation log with a timestamp other than the covered time period information is filtered in the database.

[0065] In step S322, a portion or all of the second operation log is selected as the incremental operation log. This embodiment can be understood with reference to the previous time point method. The only difference is that the backup status is determined by time period information (rather than time point information). Further detailed examples will not be provided here.

[0066] In step S230, the above incremental operation log is backed up.

[0067] Reference Figure 1As shown, for example, backup process 121 continuously reads the contents of the operation log according to set conditions, performs certain parsing and encoding verification, and then performs backup processing on the incremental operation log. The operation log can be backed up to local or external storage space 140, such as cloud storage space. The set conditions can be periodic monitoring conditions, such as preset time intervals (e.g., executing once every 5 minutes) or reaching a set capacity (executing once after the incremental data of the operation log reaches 64MB).

[0068] In step S240, if the incremental operation log backup is successful, update the attribute value of the delete lock attribute.

[0069] In some embodiments, in step S240 above, if the incremental operation log backup is successful, updating the attribute value of the delete lock attribute includes: if the incremental operation log backup is successful, determining the latest time-series value in the timestamp of the incremental operation log as the attribute value update result; and updating the attribute value of the delete lock attribute to the attribute value update result.

[0070] In some other embodiments, in step S240 above, if the incremental operation log backup is successful, updating the attribute value of the delete lock attribute includes: determining the union of the newly added coverage period of the incremental operation log and the existing coverage period of the backed-up operation log as the attribute value update result; and updating the attribute value of the delete lock attribute to the attribute value update result.

[0071] In step S250, the operation log storage space is reclaimed according to the attribute value of the deleted lock attribute.

[0072] For example, if an operation information log entry is generated at time T4 (T4 is later than T3, e.g., T4 = T3 + 50 seconds), it needs to be stored in the operation log storage space. Assuming that the remaining 1GB of operation log storage space needs to be reserved first, storing the generated operation information in the reclaimed portion of the operation log storage space is a priority strategy, provided that some of the already occupied operation log storage space can be reclaimed. If this is not possible, then the remaining operation log storage space will be requested and occupied. Therefore, it is necessary to determine which portions of the already occupied operation log storage space can be reclaimed.

[0073] At this point (time T4), assuming the data in LOG0 has been successfully backed up, the backup status of the operation log described by the delete lock attribute obtained at time T4 is: LOG0 has been backed up successfully. As a global identifier variable, the attribute value of the delete lock attribute at time T4 can be represented as the latest timestamp of the operation log in the backed-up operation log set LOG0, for example, timestamp T1. Then, the operation log storage space can be reclaimed based on this latest timestamp.

[0074] Figure 4A A detailed implementation flowchart of step S250 according to an embodiment of the present disclosure is shown schematically.

[0075] According to some embodiments of this disclosure, refer to Figure 4A As shown, when the above attribute value includes the latest timestamp of the backed-up operation log, in the above step S250, the operation log storage space is reclaimed according to the attribute value of the above-mentioned delete lock attribute, including the following steps: S411, S412, S413a and S413b.

[0076] In step S411, for the target operation log storage space to be reclaimed, the timestamp of the operation log in the target operation log storage space is obtained.

[0077] In step S412, the temporal sequence relationship between the latest value of the timestamp and the latest timestamp is determined.

[0078] In step S413a, if the above-mentioned time sequence indicates that the latest time value is earlier than or equal to the latest timestamp, the operation log in the target operation log storage space is deleted.

[0079] In step S413b, if the above time sequence indicates that the latest time value is later than the latest timestamp, the target operation log storage space is reserved.

[0080] In this embodiment, a delete lock attribute is set for the operation log. This delete lock attribute describes the backup status of the operation log in the database. The backup status adopts a sequential backup method. The attribute value of the delete lock attribute can be represented based on the latest timestamp of the backed-up operation log, which can indicate which operation logs have been backed up. This clearly indicates that only operation logs that have been backed up can be deleted, thereby realizing the reclamation of storage space. Based on the above delete lock attribute setting, a corresponding relationship is effectively established between the operation log, the latest backup status, and whether storage space can be reclaimed.

[0081] Figure 4B A detailed implementation flowchart of step S250 according to another embodiment of the present disclosure is illustrated schematically;

[0082] According to other embodiments of this disclosure, refer to Figure 4B As shown, in step S250 above, when the attribute value includes the overwrite period information of the backed-up operation log, the operation log storage space is reclaimed according to the attribute value of the deletion lock attribute, including the following steps: S421, S422, S423a, S423b and S423c.

[0083] In step S421, for the target operation log storage space to be reclaimed, the time period of the timestamp of the operation log in the target operation log storage space is obtained.

[0084] In step S422, the interval relationship between the time period of the timestamp and the coverage time period of the coverage time information is determined.

[0085] In step S423a, if the above interval relationship indicates that the time period in which the above timestamp is located is a subset or the entire set of the above covered time periods, the operation log in the above target operation log storage space is deleted.

[0086] In step S423b, if the above interval relationship indicates that there is an overlap between the time period where the above timestamp is located and the above covered time period, the operation log corresponding to the overlap time period is deleted, and the operation log in the remaining time period of the covered time period is retained.

[0087] In step S423c, if the above interval relationship indicates that there is no intersection between the time period where the above timestamp is located and the above covered time period, the above target operation log storage space is reserved.

[0088] In this embodiment, a delete lock attribute is set for the operation log. This delete lock attribute describes the backup status of the operation log in the database. The backup status can be sequential or skipped. The attribute value of the delete lock attribute can be represented based on the coverage period information of the backed-up operation log, indicating which operation logs have been backed up. This clearly indicates that only operation logs that have been backed up can be deleted, thus realizing the reclamation of storage space. Based on the above delete lock attribute setting, a corresponding relationship is effectively established between the operation log, the latest backup status, and whether storage space can be reclaimed.

[0089] In some embodiments, a Stone mechanism for segmented deletion of the Oplog is used to improve the efficiency of Oplog reclamation. A Stone is a logically separated region on the Oplog. A truncate point is used to separate the region to be deleted from other storage areas within the Oplog. In some embodiments, the truncate point corresponds to a timestamp or time period. Similar to MongoDB, other databases also employ a batch reclamation approach using multiple data segments as management units.

[0090] In embodiments including steps S210-S250, a delete lock attribute is set for the operation log. This delete lock attribute describes the backup status of the operation log in the database. The backup status indicates which operation logs have been backed up, thus clearly indicating that only operation logs that have been backed up can be deleted, thereby realizing the reclamation of storage space. Based on the above delete lock attribute setting, a corresponding relationship is effectively established between the operation log, the latest backup status, and whether storage space can be reclaimed. Thus, as the operation log is continuously generated, based on incremental backup, the incremental operation logs that need to be incrementally backed up on the basis of the already backed-up operation logs are determined by the attribute value of the current delete lock attribute. After the incremental operation log is successfully backed up, the attribute value of the delete lock is updated to obtain the latest backup status. At the same time, the reclamation of operation log storage space can be performed based on the delete lock attribute at each time point. No matter how frequently the operation log is updated, it can always be guaranteed that the operation logs to be reclaimed during the storage process are those that have already been backed up, realizing real-time complete backup of the database operation log, thereby meeting the need for data rollback to any point in time.

[0091] Figure 5 A flowchart illustrating a method for processing database logs according to another embodiment of this disclosure is shown schematically.

[0092] The database log processing method provided in this embodiment includes, in addition to the steps S210-S250 described above, the following steps: S510, S520, S530, S540, and S550. For simplification, in... Figure 5 Only steps S510 to S550 are illustrated. Steps S510 to S550 can be executed synchronously with steps S210 to S250 above, or sequentially. There is no restriction on the specific execution order, which depends on the timing of the user's operation rollback request relative to the timing of the backup.

[0093] In step S510, an operation rollback request is received, which carries location information indicating the rollback location.

[0094] The aforementioned location information may include: the point in time to which you need to revert, or the operation sequence information associated with that point in time (such as the previous operation, the two previous operations, etc.).

[0095] In step S520, the historical operation log corresponding to the above location information is queried in the above operation log storage space.

[0096] In step S530, if the historical operation log cannot be found, the historical operation log corresponding to the above location information is retrieved from the backed-up operation log.

[0097] In step S540, the data status corresponding to the above location information is determined based on the above historical operation log.

[0098] The historical operation log in step S540 can be the historical operation log queried from the backup operation log in step S530, or the historical operation log queried in step S520.

[0099] Based on the location information, the specific operation can be replayed or the corresponding data status can be obtained directly from the historical operation results.

[0100] In step S550, an operation rollback result is generated based on the above data status.

[0101] In the embodiment including steps S510~S550, during the continuous generation of operation logs, based on incremental backup, the incremental operation logs that need to be incrementally backed up on the basis of the already backed-up operation logs are determined by the attribute value of the current delete lock attribute. After the incremental operation logs are successfully backed up, the attribute value of the delete lock is updated to obtain the latest backup status. At the same time, the operation log storage space can be reclaimed based on the delete lock attribute at each time point. No matter how frequently the operation logs are updated, it can always be guaranteed that the operation logs reclaimed during the storage process are already backed up, realizing real-time complete backup of the database operation logs, and can handle user requests to roll back data to any point in time.

[0102] According to embodiments of this disclosure, the method further includes: recording the update operation of updating the attribute value of the deleted lock attribute in the operation log of the database. The database is the database corresponding to the primary storage node in a storage cluster built using one of the following architectures: master-slave mode, replica set, sharded cluster, or a combination of replica set and sharded cluster. The operation log is used for synchronization between the primary and secondary storage nodes. When the first secondary storage node switches to the primary storage node, the operation log is backed up and the operation log storage space is reclaimed from the database of the first secondary storage node.

[0103] Because operation logs are continuously synchronized between the primary and secondary storage nodes, for example through incremental or full synchronization, the secondary storage node replays the update operation logs of the synchronized primary node's oplogDeleteGuard, thus recording the latest information of the oplogDeleteGuard. After a switchover between the primary and secondary storage nodes, the first secondary storage node, acting as the primary node, can continue to cooperate with external management processes (e.g., for...) based on the synchronized oplogDeleteGuard. Figure 1 Example backup process 121) performs incremental log backup.

[0104] In some embodiments, when an external management process (e.g., backup process 121) frequently updates the oplogDeleteGuard attribute in the primary storage node 122, the oplogDeleteGuard value can be temporarily saved to a local file within the replica set node (e.g., the primary storage node) to provide immediate read and write capabilities.

[0105] A second exemplary embodiment of this disclosure provides an apparatus for database log processing.

[0106] Figure 6 A schematic block diagram of a database log processing apparatus according to an embodiment of the present disclosure is shown.

[0107] Reference Figure 6 As shown, the database log processing apparatus 600 provided in this embodiment includes: an attribute value acquisition module 601, an incremental log determination module 602, a backup module 603, an attribute value update module 604, and a storage space reclamation module 605.

[0108] The attribute value acquisition module 601 is used to acquire the attribute value of the delete lock attribute of the operation log in the database; the delete lock attribute is used to describe the backup status of the operation log in the database.

[0109] The incremental log determination module 602 is used to determine the incremental operation log of the database based on the aforementioned attribute values.

[0110] The backup module 603 is used to back up the incremental operation logs.

[0111] The aforementioned attribute value update module 604 is used to update the attribute value of the aforementioned delete lock attribute when the aforementioned incremental operation log backup is successful.

[0112] The aforementioned storage space reclamation module 605 is used to reclaim the operation log storage space based on the attribute value of the aforementioned deleted lock attribute.

[0113] According to embodiments of this disclosure, the above-described apparatus 600 further includes an attribute value recording module.

[0114] The aforementioned attribute value recording module is used to record the update operation of the attribute value of the deleted lock attribute in the operation log of the aforementioned database. The aforementioned database is the database corresponding to the primary storage node in a storage cluster built using one of the following architectures: master-slave mode, replica set, sharded cluster, or a combination of replica set and sharded cluster. The aforementioned operation log is used for synchronization between the primary and secondary storage nodes; when the first secondary storage node switches to the primary storage node, the operation log of the database on the first secondary storage node is backed up, and the operation log storage space is reclaimed.

[0115] According to embodiments of this disclosure, the apparatus 600 further includes: a rollback request receiving module, a query module, a data status determination module, and a rollback result generation module.

[0116] The aforementioned rollback request receiving module is used to receive operation rollback requests, which carry location information indicating the rollback destination.

[0117] The above query module is used to query the historical operation logs corresponding to the above location information in the above operation log storage space; if the historical operation logs cannot be found, the historical operation logs corresponding to the above location information are queried from the backup operation logs.

[0118] The aforementioned data status determination module is used to determine the data status corresponding to the aforementioned location information based on the aforementioned historical operation logs.

[0119] The regression result generation module described above is used to generate operation rollback results based on the aforementioned data status.

[0120] Any plurality of the functional modules included in the aforementioned device 600 may be combined into one module, or any one of the modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. At least one of the functional modules included in the aforementioned device 600 may be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in hardware or firmware, or in any one of software, hardware, and firmware implementations, or in a suitable combination of any of these. Alternatively, at least one of the functional modules included in the aforementioned device 600 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0121] A third exemplary embodiment of this disclosure provides an electronic device.

[0122] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present disclosure is shown.

[0123] Reference Figure 7 As shown, the electronic device 700 provided in this embodiment includes a processor 701, a communication interface 702, a memory 703, and a communication bus 704. The processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The memory 703 is used to store computer programs. When the processor 701 executes the program stored in the memory, it implements the database log processing method described above.

[0124] A fourth exemplary embodiment of this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that, when executed by a processor, implements the database log processing method as described above.

[0125] The computer-readable storage medium may be included in the device or apparatus described in the above embodiments; or it may exist independently and not assembled into the device or apparatus. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0126] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0127] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solutions provided in this disclosure comply with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0128] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0129] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing database logs, characterized in that, The method includes: Obtain the attribute value of the delete lock attribute of the operation log in the database; the delete lock attribute is used to describe the backup status of the operation log in the database. Based on the attribute values, determine the incremental operation log of the database; The incremental operation logs are backed up. If the incremental operation log backup is successful, update the attribute value of the delete lock attribute; Based on the attribute value of the deleted lock attribute, the operation log storage space is reclaimed; The backup status includes: the time information of the backed-up operation log; The attribute value of the delete lock attribute includes: the latest timestamp of the backed-up operation log; or, The attribute values ​​of the deleted lock attribute include: the overwrite period information of the backed-up operation log; The attribute value of the deleted lock is dynamically updated as the incremental operation log is backed up. Before and during the incremental operation log backup, the attribute value uses the result before the update; after the incremental operation log backup is successful, the attribute value uses the result after the update.

2. The method according to claim 1, characterized in that, If the attribute value includes the latest timestamp of the backed-up operation log, the operation log storage space is reclaimed according to the attribute value of the delete lock attribute, including: For the target operation log storage space to be reclaimed, obtain the timestamp of the operation log in the target operation log storage space; Determine the temporal relationship between the latest value of the timestamp and the latest timestamp; If the time sequence indicates that the latest time value is earlier than or equal to the latest timestamp, the operation log in the target operation log storage space will be deleted. If the chronological order indicates that the latest time value is later than the latest timestamp, the target operation log storage space is reserved.

3. The method according to claim 1, characterized in that, If the attribute value includes the overwrite period information of the backed-up operation log, the operation log storage space is reclaimed according to the attribute value of the delete lock attribute, including: For the target operation log storage space to be reclaimed, obtain the time period of the timestamp of the operation log in the target operation log storage space; Determine the interval relationship between the time period where the timestamp is located and the coverage time period of the coverage time period information; If the interval relationship indicates that the time period in which the timestamp is located is a subset or the entire set of the covered time periods, then the operation log in the target operation log storage space will be deleted. In the interval relationship, where there is an overlap between the time period where the timestamp is located and the covered time period, the operation logs corresponding to the overlapping time period are deleted, while the operation logs for the remaining time periods are retained. When the interval relationship indicates that there is no overlap between the time period where the timestamp is located and the covered time period, the storage space for the target operation log is reserved.

4. The method according to claim 1, characterized in that, Based on the attribute values, the incremental operation log of the database is determined, including: Filter the database for the first operation log whose timestamp follows the latest timestamp sequence; select part or all of the first operation log as the incremental operation log; or... Filter the database for second operation logs whose log timestamps are outside the coverage period information; select part or all of the second operation logs as the incremental operation logs.

5. The method according to claim 1, characterized in that, If the incremental operation log backup is successful, update the attribute value of the delete lock attribute, including: If the incremental operation log backup is successful, the latest time-series value in the timestamp of the incremental operation log is determined as the attribute value update result; or, the union of the newly added coverage period of the incremental operation log and the existing coverage period of the backed-up operation log is determined as the attribute value update result. Update the attribute value of the deleted lock attribute to the attribute value update result.

6. The method according to any one of claims 1-5, characterized in that, Also includes: The update operation that updates the attribute value of the deleted lock attribute is recorded in the operation log of the database; The database is the database corresponding to the primary storage node in a storage cluster built using one of the following architectures: master-slave mode architecture, replica set architecture, sharded cluster architecture, or a combination of replica set and sharded cluster architecture. The operation log is used to synchronize between the primary storage node and the slave storage node. When the first slave storage node is switched to the master storage node, the operation log backup and operation log storage space reclamation are performed on the database of the first slave storage node.

7. The method according to any one of claims 1-5, characterized in that, Also includes: Receive an operation rollback request, the operation rollback request carrying location information indicating the rollback location; Query the historical operation logs corresponding to the location information in the operation log storage space; If the historical operation logs cannot be found, retrieve the historical operation logs corresponding to the location information from the backup operation logs. Based on the historical operation logs, determine the data status corresponding to the location information; An operation rollback result is generated based on the data status.

8. A database log processing apparatus, characterized in that, include: The attribute value retrieval module is used to retrieve the attribute value of the delete lock attribute from the operation log in the database. The delete lock attribute is used to describe the backup status of the operation log in the database, and the backup status includes: the time information of when the operation log has been backed up; The attribute value of the delete lock attribute includes: the latest timestamp of the backed-up operation log; or, The attribute values ​​of the deleted lock attribute include: the overwrite period information of the backed-up operation log; The attribute value of the deleted lock is dynamically updated as the incremental operation log is backed up. Before and during the incremental operation log backup, the attribute value uses the result before the update; after the incremental operation log backup is successful, the attribute value uses the result after the update. The incremental log determination module is used to determine the incremental operation log of the database based on the attribute value. The backup module is used to back up the incremental operation logs. The attribute value update module is used to update the attribute value of the delete lock attribute when the incremental operation log backup is successful. The storage space reclamation module is used to reclaim the operation log storage space according to the attribute value of the deleted lock attribute.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1-7.

Images