System log management method and electronic device

Abstract

The application discloses a system log management method and an electronic device. The method comprises the following steps: monitoring the data change of system metadata in a storage system in real time; when detecting that the system metadata has data change, generating an incremental operation record log according to the change information of the system metadata, and storing the incremental operation record log in a system log; when a difference backup trigger condition is met, generating a difference backup record log according to a plurality of incremental operation record logs in the system log, storing the difference backup record log in the system log, and deleting the plurality of incremental operation record logs; when a snapshot trigger condition is met, generating a real-time snapshot record log according to the system metadata in the storage system, storing the real-time snapshot record log in the system log, and deleting the incremental operation record logs and the difference backup record logs in the system log, which have a generation time earlier than that of the real-time snapshot record log, to update the system log.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of system log, and in particular to a system log management method and an electronic device. BACKGROUND

[0002] The storage engine generally adopts a log-structured merge tree as its core data structure, which improves the write performance through sequential writing. Currently, a single type of operation log is usually used to record the changes of system metadata (such as adding or deleting a data file). However, for the storage engine, data changes occur in real time, which may involve frequent changes of small-scale metadata or large-scale hierarchical state changes. This will result in a large number of log entries stored in the log-structured merge tree, which not only makes the log redundant, but also increases the system burden and storage overhead. SUMMARY

[0003] The present application provides a system log management method and an electronic device to at least solve the problem that a large number of log entries are stored in the log-structured merge tree, which not only makes the log redundant, but also increases the system burden and storage overhead.

[0004] The present application provides a system log management method, comprising:

[0005] monitoring the data change of system metadata in a storage system in real time;

[0006] When detecting that the system metadata has data changes, generating an incremental operation record log according to the change information of the system metadata, and storing the incremental operation record log in a system log;

[0007] When a differential backup trigger condition is met, generating a differential backup record log according to a plurality of incremental operation record logs in the system log, storing the differential backup record log in the system log, and deleting the plurality of incremental operation record logs;

[0008] When a snapshot trigger condition is met, generating a real-time snapshot record log according to the system metadata in the storage system, and storing the real-time snapshot record log in the system log;

[0009] According to the generation time of the real-time snapshot record log, deleting the incremental operation record logs and the differential backup record logs in the system log whose generation time is earlier than the generation time of the real-time snapshot record log, to update the system log.

[0010] The present application also provides a system log management device, comprising:

[0011] An acquisition module is configured to monitor data changes of system metadata in a storage system in real time.

[0012] A processing module is configured to, when detecting that the system metadata has data changes, generate an incremental operation record log according to change information of the system metadata, and store the incremental operation record log in a system log.

[0013] The processing module is further configured to, when a differential backup trigger condition is met, generate a differential backup record log according to a plurality of incremental operation record logs in the system log, store the differential backup record log in the system log, and delete the plurality of incremental operation record logs.

[0014] The processing module is further configured to, when a snapshot trigger condition is met, generate a real-time snapshot record log according to system metadata in the storage system, and store the real-time snapshot record log in the system log.

[0015] The processing module is further configured to delete, according to a generation time of the real-time snapshot record log, incremental operation record logs and differential backup record logs in the system log that have a generation time earlier than the generation time of the real-time snapshot record log, to update the system log.

[0016] The application further provides an electronic device, including a memory configured to store a computer program, and a processor configured to implement steps of any of the system log management methods when executing the computer program.

[0017] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement steps of any of the system log management methods.

[0018] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement steps of any of the system log management methods.

[0019] Through the application, data change of system metadata in a storage system is monitored in real time; when it is detected that there is data change of the system metadata, an incremental operation record log is generated according to change information of the system metadata, and the incremental operation record log is stored in a system log; when a differential backup triggering condition is met, a differential backup record log is generated according to a plurality of incremental operation record logs in the system log, the differential backup record log is stored in the system log, and the plurality of incremental operation record logs are deleted; when a snapshot triggering condition is met, a real-time snapshot record log is generated according to system metadata in the storage system, the real-time snapshot record log is stored in the system log; and according to a generation time of the real-time snapshot record log, incremental operation record logs and differential backup record logs in the system log whose generation times are earlier than the generation time of the real-time snapshot record log are deleted to update the system log. In the scheme, by setting the forms of the incremental operation record log, the differential backup record log and the real-time snapshot record log, the corresponding record logs can be flexibly generated and stored according to data change, and by the association among the incremental operation record log, the differential backup record log and the real-time snapshot record log, redundant record logs can be deleted in time, so that the number of the system log can be effectively reduced, the storage space is saved, and the system burden and storage overhead are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly explain the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0021] Figure 1 A system log management method provided by an embodiment of the present application Figure 1

[0022] Figure 2 A system log management method provided by an embodiment of the present application Figure 2

[0023] Figure 3 A system log management method provided by an embodiment of the present application Figure 3

[0024] Figure 4 A system log management method provided by an embodiment of the present application Figure 4

[0025] Figure 5 A system log management device provided by an embodiment of the present application

[0026] ​​​​Figure 6 A structural diagram of an electronic device is provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0028] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0029] It should be noted that, in the embodiments of the present application, the words “exemplary” or “for example” are used to represent an example, illustration or description. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words “exemplary” or “for example” are intended to present the relevant concept in a specific manner.

[0030] Modern high-performance key-value storage engines (such as LevelDB, RocksDB, etc.) generally use a Log-Structured Merge-Tree (LSM-Tree) as their core data structure. The LSM-Tree is a data structure designed for efficient processing of write-intensive scenarios, especially suitable for NoSQL databases. Its core idea is to transform random writes into sequential writes, significantly improving write performance, while balancing read-write efficiency through hierarchical storage and background merging mechanisms, but its metadata management faces serious challenges. Metadata usually includes current valid log files, sequence numbers, table file (SSTable) information of each data level and its key attributes (such as key range, file size, creation time, etc.).

[0031] To ensure the system can recover to a consistent state after a crash, the persistence of metadata is crucial. Existing technologies (such as Manifest of RocksDB) usually use a single type of operation log (Operation Log) to record the changes of metadata (such as adding or deleting an SSTable). When the system starts, it needs to replay the entire operation log from beginning to end to reconstruct the final metadata state. This reconstruction has two significant defects: first, when the log file becomes very large, the recovery process will be very slow due to the need to traverse all the operation logs, seriously affecting the startup speed of the system and the availability of services; second, the single log mode lacks flexibility. For some small-scale metadata that changes frequently (such as the progress marker of compression operations), recording complete file information is redundant; while for large-scale state changes (such as the file list change of the entire level after a major compression), recording the add or delete operation of each file one by one will generate a large number of log entries, increasing I / O burden and storage overhead.

[0032] In addition, existing snapshot mechanisms usually only save the full state of metadata and cannot cope with the scenario of frequent updates of part of the metadata. When the system needs to quickly roll back to a certain state, the recovery efficiency of the full snapshot is low, and it occupies a large amount of storage space. Therefore, there is currently a need for an efficient and flexible metadata management solution that can speed up the recovery process after a system crash and adapt to different metadata change scenarios.

[0033] To address the aforementioned technical issues, this application provides a system log management method and electronic device that monitors real-time changes in system metadata within a storage system. When a change in system metadata is detected, an incremental operation log is generated based on the change information and stored in the system log. When a differential backup trigger condition is met, a differential backup log is generated based on multiple incremental operation logs in the system log, stored in the system log, and multiple incremental operation logs are deleted. When a snapshot trigger condition is met, a real-time snapshot log is generated based on the system metadata within the storage system and stored in the system log. Based on the generation time of the real-time snapshot log, incremental operation logs and differential backup logs whose generation time is earlier than the generation time of the real-time snapshot log are deleted from the system log to update the system log. This solution allows for the flexible generation and storage of corresponding logs based on data changes by setting up three types of logs: incremental operation logs, differential backup logs, and real-time snapshot logs. Furthermore, by establishing relationships between these three types of logs, redundant logs can be promptly deleted. This effectively reduces the number of system logs, saves storage space, and lowers system load and storage overhead.

[0034] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] like Figure 1 As shown, Figure 1 A flowchart of a system log management method provided for embodiments of this application, the method may include the following steps:

[0036] 101. Monitor the changes in system metadata in the storage system in real time.

[0037] In this embodiment of the application, the storage system stores a lot of system metadata. As the server or device runs normally, the system metadata will continue to change, such as adding data, modifying data, deleting data, compressing data, etc. The changes of various data in the system metadata can be recorded in the form of LSM-Tree. Therefore, it is necessary to monitor the data changes of the system metadata in real time.

[0038] 102. When a data change is detected in the system metadata, an incremental operation log is generated based on the change information of the system metadata, and the incremental operation log is stored in the system log.

[0039] In the embodiments of the present application, if it is detected that there is data change in the system metadata, that is, the data stored in the current storage system has changed, in order to record this data change, an incremental operation record log can be generated according to the change information of the system metadata, the incremental operation record log can describe the change information of the specific system metadata, and the incremental operation record log is stored in a system log, and the system log can be used to store various logs, and each log can record at least one data change of the storage system.

[0040] For example, assuming that a user adds a data A in the storage system, at this time, it is detected that there is data change in the system metadata, and the change needs to be recorded. Specifically, it can be recorded that a data A is newly stored in the storage system, and the data A is stored in the cache partition of the storage system, so that it is known that a data A is stored in the cache partition at the current time when subsequent data rollback or system reconstruction is performed.

[0041] In some embodiments, the incremental operation record log can record discrete and fine-grained change operations on the system metadata. That is, the incremental operation record log focuses on every small and independent change of the system metadata in the running process. The incremental operation record log is suitable for scattered and small-scale metadata update scenarios. For example, when a MemTable flush generates a new SSTable file, there will be corresponding system metadata changes, such as addition of a new file, and the like, which can be recorded by the incremental operation record log.

[0042] Each incremental operation record log contains an operation type, a target level, a file identifier, and related attributes. The operation type can be used to explicitly indicate an AddFile, DeleteFile, or the like. The target level can be used to indicate in which hierarchical structure the operation occurs. The file identifier can be used to uniquely determine the involved file. The related attributes, such as for the AddFile operation, record the level of the file, the file identifier (file_id), the smallest key (smallest_key) and the largest key (largest_key) of the file (used to define the data range contained by the file), the file size (file_size), the checksum (checksum, used to verify the integrity of the file), and the like. For the DeleteFile operation, the level of the file and the file identifier are mainly recorded. For example: AddFile(level, file_id,smallest_key, largest_key, file_size, checksum) or DeleteFile(level, file_id).

[0043] The incremental operation record log records each minor change of the system metadata in detail, so as to track the change history of the system metadata. When it is necessary to analyze the evolution process of the system metadata or to perform fine data recovery, the incremental operation record log can provide detailed operation information, and help to restore to a specific system data state.

[0044] In some embodiments, for the data change of the system metadata, each time can involve more than one data change, and a plurality of files can be simultaneously batched and stored or deleted. When the incremental operation record log is generated, it can be generated according to each bottommost data change. That is, one incremental operation record log is generated for one data change, and as many incremental operation record logs as the data changes are generated, and all are stored in the system log.

[0045] 103、When the differential backup triggering condition is met, a differential backup record log is generated according to the plurality of incremental operation record logs in the system log, the differential backup record log is stored in the system log, and the plurality of incremental operation record logs are deleted.

[0046] In the embodiments of the present application, since one incremental operation record log is generated for each data change, if there are a large number of data changes in the system metadata, a large number of incremental operation record logs will be stored in the system log, which greatly increases the storage burden. Therefore, the plurality of incremental operation record logs stored in the system log can be summarized to generate a differential backup record log. The differential backup record log can be understood as a summary log of the plurality of incremental operation record logs in the system log. Therefore, after the differential backup record log is generated according to the plurality of incremental operation record logs, the plurality of incremental operation record logs can be deleted. That is, one differential backup record log has summarized the specific system metadata change described by the plurality of incremental operation record logs. According to the differential backup record log, the system metadata change corresponding to the plurality of incremental operation record logs can be known. At this time, only one differential backup record log can be stored in the system log, and the plurality of incremental operation record logs can be deleted, so as to reduce the storage amount of the system log.

[0047] It should be noted that when the difference backup record log is generated according to the plurality of incremental operation record logs in the system log, the corresponding difference backup triggering condition can be set, and the difference backup record log is not generated at will. Since the difference backup record log can be simply understood as a summary of the plurality of incremental operation record logs, it can be thought that the difference backup triggering condition is set according to the number of incremental operation record logs. Of course, since the incremental operation record log is a record log for each specific data change, it cannot reflect the hierarchical relationship between the data, and the storage system can be set to have multiple levels. If data changes occur between different levels, the structure and association between the levels cannot be recorded directly through the incremental operation record log. Therefore, the difference backup triggering condition can also be set according to the data change between the levels of the storage system. Of course, the difference backup triggering condition can also be set according to other change conditions, and the embodiments of the present application are not limited in this regard.

[0048] In some embodiments, the difference backup record log (Diff Backup Record) can be understood as a record of a relatively complete change set on a certain specific subsystem or dimension. The difference backup record log is not simply recording all single change operations between two snapshots, but from a more macroscopic perspective, recording the overall change of a specific part. For example: after a large-scale compression (Compaction) is completed, a difference backup record can be generated according to all data changes involved in this compression. The compression operation usually involves the merging, deletion and reorganization of multiple files, and will generate a large amount of system metadata changes.

[0049] The difference backup record log does not list all the added and deleted file operations, but directly records the complete file list of the target level involved in this compression after completion (or the change summary of this level relative to the last snapshot). The difference backup record log specifically includes the identification of the affected level (which level has changed), the new file list (listing the newly added files in this level after compression), the deleted file list (listing the deleted files), and related metadata such as the new key range (reflecting the new range of data contained in the file after compression), file size statistics (statistics of the size of the file after compression), etc.

[0050] The difference backup record log avoids a large number of scattered incremental records for a single large-scale change. When facing operations such as large-scale compression that will generate a large amount of system metadata changes, if the incremental operation record log is used, a large number of records will be generated, which is complex to manage and inefficient. The difference backup record log records the change set as a whole, significantly improves the log efficiency, reduces the storage amount and management difficulty of the log, and also facilitates quick recovery and analysis of large-scale changes.

[0051] 104. When the snapshot trigger condition is met, a real-time snapshot record log is generated according to the system metadata in the storage system, and the real-time snapshot record log is stored in the system log.

[0052] In the embodiments of the present application, in order to comprehensively prepare the system metadata in the record storage system, in addition to the incremental operation record log and the differential backup record log described above, a real-time snapshot record log can also be generated. The real-time snapshot record log can be understood as a full record of all system metadata in the storage system at the current time, that is, the current all system metadata is recorded in the form of a snapshot. A real-time snapshot record log is generated, which records all system metadata information such as all valid data files, serial numbers, and hierarchical structures at the time when the real-time snapshot record log is generated. It can be popularly understood that a picture of the current storage system is taken, which shows all the data of the storage system. Then the real-time snapshot record log can be stored in the system log.

[0053] It should be noted that the real-time snapshot record log can be generated according to the system metadata in the storage system, and the corresponding snapshot trigger condition can be set. The real-time snapshot record log is not generated randomly. As time goes on, the system metadata will change more and more, and the incremental operation record log and the differential backup record log stored in the system log will also increase. Therefore, it can be thought that the snapshot trigger condition can be set according to the number of record logs stored in the system log. Of course, since the real-time snapshot record log can be a record of all system metadata at the current time, it is necessary to consume certain resources to generate the real-time snapshot record log. Therefore, it is not suitable to generate the snapshot record frequently. Therefore, a time interval can be set as the snapshot trigger condition according to the time when the last real-time snapshot record log is generated. Of course, the snapshot trigger condition can also be set according to other conditions, which are not limited in the embodiments of the present application.

[0054] In some embodiments, the real-time snapshot record log (Snapshot Record) is a complete metadata state mirror generated periodically. This means that the system will take a comprehensive "picture" of the state of the entire system metadata according to a specific trigger condition, and record the state of the system metadata at this time completely.

[0055] The real-time snapshot record log records all valid data file related information at the time of generating the real-time snapshot record log, including data files themselves, serial numbers (used to uniquely identify file versions, etc.), hierarchy (organization hierarchy of data in the system), and all system metadata. The real-time snapshot record log can be stored in an efficient binary encoding format, which helps to save storage space and improve read / write efficiency, and specifically includes a file list (listing all related files), a hierarchy configuration (describing the distribution and relationship of data at different levels), version information (recording the version of the data), and the like.

[0056] The real-time snapshot record log can be used as a reference point for system recovery. When the system fails or needs to be restored to a certain state, the system can be restored to the state at the time of generating the real-time snapshot record log according to the complete view of system metadata provided by the real-time snapshot record log. For example, after a database system crashes, the system metadata structure of the database can be quickly rebuilt by loading the latest real-time snapshot record log, providing a basis for subsequent data recovery.

[0057] 105. According to the time of generating the real-time snapshot record log, the incremental operation record logs and the differential backup record logs in the system log generated before the time of generating the real-time snapshot record log are deleted to update the system log.

[0058] In the embodiments of the present application, since the real-time snapshot record log records all data in the storage system at the time of generating the real-time snapshot record log, the incremental operation record logs and the differential backup record logs generated and stored before the time of generating the real-time snapshot record log only record the change information of the data before the time of generating the real-time snapshot record log. It can be understood that the real-time snapshot record log includes the data change information described in the incremental operation record logs and the differential backup record logs stored in the system log before the real-time snapshot record log is generated, so that the system reconstruction can be implemented based on all data of the storage system described in the real-time snapshot record log when the system is reconstructed, and the data change information before the time of generating the real-time snapshot record log is no longer needed. In order to further save the storage space of the system log and reduce the amount of logs, the incremental operation record logs and the differential backup record logs in the system log generated before the time of generating the real-time snapshot record log can be deleted. That is, after the real-time snapshot record log is generated, the real-time snapshot record log can be immediately stored in the system log, and all incremental operation record logs and differential backup record logs generated before the time of generating the real-time snapshot record log in the system log can be deleted. In this way, at least one real-time snapshot record log can be saved in the system log, and the system log is updated.

[0059] Exemplarily, the complete process of the system log management method provided in the embodiments of the present application can include: a data A is newly added to the system metadata, and a corresponding incremental operation record log 1 is generated and stored in the system log; at the next moment, a data B is deleted from the system metadata, and a corresponding incremental operation record log 2 is generated and stored in the system log; and so on. The incremental operation record logs stored in the system log are more and more, until it is detected that the differential backup triggering condition is triggered. At this moment, 300 incremental operation record logs are stored in the system log, and then the 300 incremental operation record logs can be summarized to generate a differential backup record log 1 and store the differential backup record log 1 in the system log, and the 300 incremental operation record logs are deleted. At this moment, only the differential backup record log 1 is stored in the system log. Then, the system metadata continues to change, and the system log continues to store new incremental operation record logs. When the differential backup triggering condition is triggered again, a differential backup record log 2 is generated and stored in the system log. Until it is detected that the snapshot triggering condition is triggered, it is assumed that the differential backup record logs 1, 2, 3 and 4 and 58 incremental operation record logs are stored in the system log at this moment. Then, the real-time snapshot record log 1 can be generated based on all the system metadata in the current storage system and stored in the system log, and then the differential backup record logs 1, 2, 3 and 4 and the 58 incremental operation record logs stored in the system log are deleted. At this moment, only the real-time snapshot record log 1 can be retained in the system log, and of course, the real-time snapshot record log generated at an earlier moment can also be retained, and the incremental operation record logs and the differential backup record logs are not saved.

[0060] In some embodiments, when the incremental operation record logs and the differential backup record logs generated at an earlier moment than the generation moment of the real-time snapshot record log are deleted from the system log according to the generation moment of the real-time snapshot record log, the historical snapshot record log generated at a historical moment can also be deleted. In this way, only the real-time snapshot record log generated at the current moment is stored in the system log. Of course, the historical snapshot record log can also be retained. In this way, the multiple snapshot record logs can be stored in the system log, and the incremental operation record logs and the differential backup record logs are not saved. In this way, the fault tolerance of the subsequent system reconstruction can be increased, and the historical state can be rolled back and queried based on the real-time snapshot record log generated at the current moment.

[0061] In the embodiments of the present application, by setting the forms of the incremental operation record log, the differential backup record log and the real-time snapshot record log, the corresponding record logs can be flexibly generated and stored according to the data change situation. At the same time, by the association among the incremental operation record log, the differential backup record log and the real-time snapshot record log, the redundant record logs can be timely deleted. In this way, the number of the system logs can be effectively reduced, the storage space is saved, and the system burden and the storage overhead are reduced.

[0062] As shown in Figure 2 , another flowchart of the system log management method provided by the embodiments of the present application can include the following steps: Figure 2

[0063] 201. When the storage system is initialized, an initial snapshot record log is created.

[0064] In the embodiments of the present application, when the storage system is started, initialization is needed, and at the time of initialization, the system log can or can not store a record log, that is, the system log is empty, and at this time, an initial snapshot record log can be created regardless of whether the record log exists or not. The initial snapshot record log is used to record the system metadata in the storage system at the time of initialization, and if the system metadata exists at the time of initialization in the storage system, the initial snapshot record log can record the information of the corresponding system metadata; if the system metadata does not exist at the time of initialization in the storage system, the initial snapshot record log can be an empty record log.

[0065] 202. The initial snapshot record log is stored to generate a system log.

[0066] In the embodiments of the present application, after the initial snapshot record log is created, the initial snapshot record log needs to be stored, that is, it needs to be stored through the system log, and therefore, the system log can be generated on the basis of the initial snapshot record log. The system log currently only stores the initial snapshot record log, and in the future, with a large amount of data change, the system log will also store the incremental operation record log, the differential backup record log and the real-time snapshot record log and the like.

[0067] 203. The data change of the system metadata in the storage system is monitored in real time.

[0068] In the embodiments of the present application, for the description of step 203, please refer to the detailed description of step 101 in the above embodiments, and the embodiments of the present application will not be described herein.

[0069] 204. When it is detected that the system metadata has multiple data changes, the data amount of the changed data is detected.

[0070] 205. If the data amount is less than a preset data amount, the change information of the system metadata corresponding to the multiple data changes is combined to generate an incremental operation record log, and the incremental operation record log is stored in the system log.

[0071] ​In the embodiments of the present application, during the running of the storage system, data changes of system metadata occur in real time, and there can be many data changes at the same time, such as adding data A, deleting data B, adjusting the parameter value in data C to 1, compressing data D and data E to obtain data F, and the above data changes can occur at the same time, and then 4 incremental operation record logs need to be generated to record the above 4 data changes, which is relatively cumbersome, therefore, the data change information occurring at the same time can be combined to generate an incremental operation record log to record, but because the size of the incremental operation record log is limited, too much data change information cannot be recorded at one time, therefore, if it is detected that there are multiple data changes of system metadata at a moment, the data volume of the data that is changed can be determined, if the data volume is large, the incremental operation record log cannot be recorded by one incremental operation record log, then the incremental operation record log of each data change is still generated respectively; and if the data volume is less than a preset data volume, it is indicated that the data volume involved in the multiple data changes does not exceed the data volume upper limit corresponding to one incremental operation record log, therefore, the change information of the system metadata involved in the multiple data changes can be combined to generate an incremental operation record log, and the incremental operation record log is stored into the system log.

[0072] In the embodiments of the present application, when the incremental operation record log is generated, if the data volume involved in the multiple data changes is small, the multiple system metadata changes can be combined into one incremental operation record log for recording and storage, which can further reduce the log amount stored, and the consistency of the multiple data changes can be reflected.

[0073] 206、When it is detected that the number of incremental operation record logs in the system log reaches the first number, it is determined that the differential backup triggering condition is met.

[0074] In the embodiments of the present application, because the differential backup record log can be simply understood as the summary of multiple incremental operation record logs, it can be thought that the differential backup triggering condition is set according to the number of incremental operation record logs, that is, if a certain number of incremental operation record logs have been stored in the system log, the number is large, and the multiple incremental operation record logs need to be summarized and aggregated, therefore, according to the multiple incremental operation record logs in the system log, the differential backup record log can be generated when it is detected that the number of incremental operation record logs in the system log reaches the first number, that is, the differential backup triggering condition is met.

[0075] In some embodiments, the first number can be a value set by the user, can be determined according to historical experience, can be determined according to the maximum storage amount of the system log, can be set in other manners, and the embodiments of the present application are not limited specifically.

[0076] 207. When it is detected that there is a storage system level change in the plurality of incremental operation record logs stored in the system log, it is determined that the differential backup trigger condition is met.

[0077] In the embodiments of the present application, since the incremental operation record log is a record log for each data change, it cannot reflect the hierarchical relationship between the data, and the storage system has multiple levels. If there is a data change between different levels, only the incremental operation record log cannot intuitively record the structure and association between the levels. Therefore, the differential backup trigger condition can be set according to the data change between the levels of the storage system, that is, if there is a level change of the storage system in the data change indicated by the plurality of incremental operation record logs, for example, 500 data in the L0 level are compressed to obtain 10 data, and the 10 data are stored in the L1 level, and the 500 data in the L0 level are deleted. At this time, it can be seen that the L0 level and the L1 level of the storage system are changed, and therefore it can be considered that the differential backup trigger condition is met. Therefore, the specific change information of the 500 data in the L0 level and the 10 data in the L1 level needs to be recorded by a differential backup record log.

[0078] In the embodiments of the present application, the steps 206 and 207 are two different detection of the differential backup trigger condition. The steps 206 and 207 can be an and / or relationship, that is, as long as the number of incremental operation record logs in the system log reaches the first number, it can be considered that the differential backup trigger condition is met. Similarly, as long as there is a storage system level change in the plurality of incremental operation record logs stored in the system log, it can be considered that the differential backup trigger condition is met. Of course, it can also be that the number of incremental operation record logs in the system log reaches the first number, and at the same time, there is a storage system level change in the plurality of incremental operation record logs stored in the system log. In this case, it can also be considered that the differential backup trigger condition is met.

[0079] In the embodiments of the present application, since the differential backup record log can be simply understood as a summary of a plurality of incremental operation record logs, it can be thought that the differential backup triggering condition is set according to the number of incremental operation record logs. Of course, since the incremental operation record log is a record log for each data change, it cannot reflect the hierarchical relationship between the data, and a plurality of levels are set in the storage system. If the data change between different levels occurs, the structure and association between the levels cannot be recorded intuitively only by the incremental operation record log. Therefore, the differential backup triggering condition can also be set according to the data change between the levels of the storage system. In this way, the plurality of incremental operation record logs can be summarized to generate the differential backup record log when the incremental operation record log and / or the level change are involved, the system log storage amount is effectively saved, and the data changes of various types are intuitively displayed.

[0080] 208. When the differential backup triggering condition is met, the differential backup record log is generated according to the plurality of incremental operation record logs in the system log, the differential backup record log is stored in the system log, and the plurality of incremental operation record logs are deleted.

[0081] In the embodiments of the present application, for the description of step 208, refer to the detailed description of step 103 in the above embodiments, and the embodiments of the present application will not be repeated.

[0082] 209. When it is detected that the amount of record logs stored in the system log reaches the second number, and / or the time interval between the snapshot record log stored in the system log reaches the preset time length, it is determined that the snapshot triggering condition is met.

[0083] In the embodiments of the present application, as time goes on, the change of the system metadata will be more and more, and the incremental operation record log and the differential backup record log stored in the system log will also be more and more. Therefore, it can be thought that the snapshot triggering condition is set according to the number of record logs stored in the system log, that is, if it is detected that the amount of record logs stored in the system log reaches the second number, it can be considered that the snapshot triggering condition is met, and then the real-time snapshot record log is generated. Of course, since the real-time snapshot record log can be a record of all system metadata at the current time, it is necessary to consume certain resources to generate the real-time snapshot record log, and it is not suitable to generate the snapshot record frequently. Therefore, a time interval can be set as the snapshot triggering condition according to the time when the last real-time snapshot record log is generated, that is, if it is detected that the time interval between the snapshot record log stored in the system log reaches the preset time length, it can be understood that the generation of the snapshot record log is set to a corresponding period, and the snapshot record log is generated according to the period, and it is considered that the snapshot triggering condition is met.

[0084] In some embodiments, the second quantity can be a value set by the user, can be determined according to historical experience, can be determined according to the maximum storage amount of the system log, or can be set in other manners, and the application embodiments do not make specific limitations. The second quantity and the first quantity can not have a specific size relationship. In addition, the preset time length can also be a value set by the user, or can be determined according to historical experience, and the application embodiments do not make specific limitations.

[0085] In some embodiments, as can be seen from the above description, the differential backup trigger condition and the snapshot trigger condition can both be self-set conditions, and of course can both be determined according to the log quantity, or can be determined as other conditions. It should be noted that the generation of the real-time snapshot record log and the generation of the differential backup record log can not have a specific sequence. Since the differential backup record log can be a summary of multiple incremental operation record logs, the differential backup record log is generated according to multiple incremental operation record logs, and the real-time snapshot record log is a full record of all system metadata currently stored and does not care about data changes. Therefore, in combination with the differential backup trigger condition and the snapshot trigger condition, the following situations can occur:

[0086] Case one: After generating and storing multiple incremental operation record logs, a differential backup record log is generated and stored according to the differential backup trigger condition that the number of incremental operation record logs reaches the first quantity; then multiple incremental operation record logs are continuously generated and stored, and a real-time snapshot record log is generated and stored according to the snapshot trigger condition that the time interval between the snapshot record log stored in the system log and the real-time snapshot record log reaches the preset time length.

[0087] Case two: After generating and storing multiple incremental operation record logs, a real-time snapshot record log is generated and stored according to the snapshot trigger condition that the time interval between the snapshot record log stored in the system log and the real-time snapshot record log reaches the preset time length; then multiple incremental operation record logs are continuously generated and stored, and a differential backup record log is generated and stored according to the differential backup trigger condition that the number of incremental operation record logs reaches the first quantity.

[0088] Case three: After generating and storing multiple incremental operation record logs, a differential backup record log is generated and stored according to the differential backup trigger condition that there is a storage system level change in the multiple incremental operation record logs; then multiple incremental operation record logs are continuously generated and stored, and a real-time snapshot record log is generated and stored according to the snapshot trigger condition that the time interval between the snapshot record log stored in the system log and the real-time snapshot record log reaches the preset time length.

[0089] Case four: after generating and storing a plurality of incremental operation record logs, a real-time snapshot record log is generated and stored according to a snapshot trigger condition that a time interval between the snapshot record log stored in the system log and the plurality of incremental operation record logs reaches a preset time length; then a plurality of incremental operation record logs are continuously generated and stored, and a differential backup record log is generated and stored according to a differential backup trigger condition that there is a storage system level change in the plurality of incremental operation record logs.

[0090] Case five: after generating and storing a plurality of incremental operation record logs, a differential backup record log is generated and stored according to a differential backup trigger condition that there is a storage system level change in the plurality of incremental operation record logs; then a plurality of incremental operation record logs are continuously generated and stored, and a real-time snapshot record log is generated and stored according to a snapshot trigger condition that a record log amount stored in the system log reaches a second quantity.

[0091] Case six: after generating and storing a plurality of incremental operation record logs, a real-time snapshot record log is generated and stored according to a snapshot trigger condition that a record log amount stored in the system log reaches a second quantity; then a plurality of incremental operation record logs are continuously generated and stored, and a differential backup record log is generated and stored according to a differential backup trigger condition that there is a storage system level change in the plurality of incremental operation record logs.

[0092] Case seven: when triggering according to a differential backup trigger condition that a quantity of incremental operation record logs reaches a first quantity and a snapshot trigger condition that a record log amount stored in the system log reaches a second quantity, the differential backup record log and the real-time snapshot record log can be respectively generated according to a size relationship between the first quantity and the second quantity.

[0093] In the embodiment of the present application, as time goes on, the change of system metadata will be more and more, and the incremental operation record logs and the differential backup record logs stored in the system log will also be more and more, so it can be thought that the snapshot trigger condition can be set according to the quantity of record logs stored in the system log; of course, since the real-time snapshot record log can be a record of all system metadata at the current time, it is necessary to consume certain resources to generate the real-time snapshot record log, and it is not suitable to generate the snapshot record frequently, so a time interval can be set as the snapshot trigger condition according to a time of generating the real-time snapshot record log last time, so that the real-time snapshot record log can be generated immediately when the above condition is met, the system metadata is recorded in full, the system log storage amount is effectively saved, and a data recovery benchmark is provided for subsequent system reconstruction.

[0094] 210、when the snapshot trigger condition is met, generating a real-time snapshot record log according to system metadata in the storage system, and storing the real-time snapshot record log in the system log.

[0095] 211、according to a generation time of the real-time snapshot record log, deleting, from the system log, an incremental operation record log and a differential backup record log whose generation time is earlier than the generation time of the real-time snapshot record log, to update the system log.

[0096] In the embodiments of the present application, for the description of steps 210-211, refer to the detailed description of steps 104-105 in the above embodiments, and the embodiments of the present application will not be repeated.

[0097] 212、in response to a reconstruction instruction of the storage system, reconstructing the system according to the record logs stored in the system log.

[0098] In the embodiments of the present application, if the storage system has crashed and all the stored system metadata is lost, the storage system needs to be reconstructed, and when the system is reconstructed, the record logs stored in the system log can be used as a reference to reconstruct the system, and the record logs at least include the real-time snapshot record log.

[0099] In some embodiments, the reconstruction of the storage system can be implemented based on a reconstruction instruction, which can be an instruction directly issued by a user, or an instruction generated according to an operation on the storage system, to instruct the storage system to reconstruct.

[0100] It should be noted that, since the incremental operation record log and / or the differential backup record log are all records of the change information of the system metadata, and the real-time snapshot record log records the full state of all the system metadata, that is, through the real-time snapshot record log, all the system metadata stored in the storage system at the time when the real-time snapshot record log is generated can be known, and the incremental operation record log and / or the differential backup record log can be understood as subsequent data changes based on the real-time snapshot record log, so at least the real-time snapshot record log can be used to reconstruct all the system metadata corresponding to the generation time of the real-time snapshot record log.

[0101] In some embodiments, since the storage system is in normal operation, after the real-time snapshot record log is generated, the incremental operation record logs and the differential backup record logs in the system log generated at a time earlier than the generation time of the real-time snapshot record log are deleted, so it can be considered that at least one real-time snapshot record log is stored in the system log, and of course if the historical snapshot record logs are not deleted, the system log can also store the historical snapshot record logs, so when the system is reconstructed, the latest generated and stored snapshot record log can be selected for reconstruction, that is, in response to the reconstruction instruction of the storage system, the system is reconstructed according to the record log stored in the system log, which can specifically include: traversing the system log according to the generation time of each record log stored in the system log to determine at least one snapshot record log; and reconstructing the storage system according to the snapshot record log with the smallest difference between the generation time and the current time among the at least one snapshot record log.

[0102] It should be noted that each record log stored in the system log will be stored in the system log after being generated, and the generation time corresponding to the stored record log, the smaller the difference between the generation time of a record log and the current time, the later the generation time of the record log, which can also be understood as the record log being newer and being able to show the latest data change of the system metadata, so when the system is reconstructed, the system log can be traversed from back to front according to the generation time of each record log, that is, the latest record log is traversed first, and then the generation time is sequentially traversed from back to front, so as to determine at least one snapshot record log stored in the system log.

[0103] Among them, since the incremental operation record logs and the differential backup record logs in the system log generated at a time earlier than the generation time of the real-time snapshot record log are deleted when each snapshot record log is generated, it can be considered that the system log includes only the real-time snapshot record log and at least one historical snapshot record log except the incremental operation record logs and the differential backup record logs generated at a time later than the generation time of the latest real-time snapshot record log, so after the system log is traversed, at least one snapshot record log can be determined, and then the latest snapshot record log can be selected according to the generation time of each snapshot record log, that is, the snapshot record log with the smallest difference between the generation time and the current time, to reconstruct the storage system.

[0104] For example, assuming that the snapshot trigger condition for generating the snapshot record log is to generate one every two hours, then at the initialization of the storage system at 0 o'clock every day, snapshot record log 1 can be generated and stored, and at 2 o'clock, snapshot record log 2 can be generated and stored, at which time all the incremental operation record logs and differential backup record logs generated during the period from 0 o'clock to 2 o'clock can be deleted, and the system log only retains snapshot record log 1 and snapshot record log 2; continue to generate and store snapshot record log 3 at 4 o'clock, and delete all the incremental operation record logs and differential backup record logs generated during the period from 2 o'clock to 4 o'clock, and the system log only retains snapshot record log 1, snapshot record log 2 and snapshot record log 3; and so on. Assuming that the storage system crashes at 7:30, at which time the storage system needs to be reconstructed, according to the snapshot trigger condition, it can be known that the system log stores snapshot record log 1, snapshot record log 2, snapshot record log 3 and snapshot record log 4, and all the incremental operation record logs and differential backup record logs generated during the period from 6 o'clock to 7:30; therefore, the at least one snapshot record log determined by traversing the system log can include snapshot record log 1, snapshot record log 2, snapshot record log 3 and snapshot record log 4. Since the generation time of snapshot record log 4 is 6 o'clock, the difference between the current time (7:30) and the generation time of snapshot record log 4 is the smallest, that is, snapshot record log 4 is the latest snapshot record log, and therefore the storage system can be reconstructed according to snapshot record log 4.

[0105] In some embodiments, although the real-time snapshot record log records the full state of all system metadata, and according to the real-time snapshot record log, it can be known which system metadata is included in the current storage system, the real-time snapshot record log is not continuously generated, but is generated only when the snapshot trigger condition is met, and therefore between the generation of adjacent two snapshot record logs, data changes of the storage system also occur, which need to be recorded through the incremental operation record log and the differential backup record log. If the storage system crashes between the generation of adjacent two snapshot record logs, only the latest generated snapshot record log is used for reconstruction, although it can be considered that a large proportion of system metadata is recovered, but the data changes of the storage system occurring after the generation of the snapshot record log cannot be recovered, and therefore on the basis of the latest generated snapshot record log, the storage system can also be reconstructed in combination with the incremental operation record log and / or the differential backup record log, that is, according to the snapshot record log with the smallest difference between the generation time and the current time among the at least one snapshot record log, the storage system is reconstructed, and specifically, it can include: if the system log still exists the incremental operation record log and / or the differential backup record log, then according to the snapshot record log with the smallest difference between the generation time and the current time among the at least one snapshot record log, and the incremental operation record log and / or the differential backup record log, the storage system is reconstructed.

[0106] It should be noted that, since the incremental operation record log and the differential backup record log in the system log whose generation time is earlier than that of the snapshot record log are deleted when each snapshot record log is generated, it can be thought that, if the incremental operation record log and / or the differential backup record log exist in the system log, the generation time of the incremental operation record log and / or the differential backup record log must be later than that of the real-time snapshot record log, that is, the incremental operation record log and / or the differential backup record log records the data change of the system metadata which occurs on the basis of all the system metadata described in the latest snapshot record log, and thus the storage system can be reconstructed according to the snapshot record log whose difference between the generation time and the current time is the smallest among the at least one snapshot record log, and the incremental operation record log and / or the differential backup record log.

[0107] For example, it is assumed that the snapshot trigger condition for generating the snapshot record log is to generate once every two hours, and various record logs are continuously generated according to the change of the system metadata from the initialization of the storage system at 0 o'clock, and it is assumed that the storage system crashes at 7:30, and according to the snapshot trigger condition, it can be known that the snapshot record log 1, the snapshot record log 2, the snapshot record log 3 and the snapshot record log 4, and all the incremental operation record logs and the differential backup record logs generated during the period from 6 o'clock to 7:30 are stored in the system log; when the storage system is reconstructed, since the generation time of the snapshot record log 4 is 6 o'clock, the difference between the generation time and the current time (7:30) is the smallest, that is, the snapshot record log 4 is the latest snapshot record log, and if the system is reconstructed only by the snapshot record log 4, it can be thought that the reconstructed storage system can only be restored to the system metadata state at 6 o'clock, and various data changes occurring after 6 o'clock cannot be restored, and thus the snapshot record log 4 can be used as a data recovery reference, and data recovery is continued according to all the incremental operation record logs and the differential backup record logs generated during the period from 6 o'clock to 7:30 on the basis of the data recovery according to the snapshot record log 4, so that the system metadata of the storage system can be restored to the system metadata state at 7:30 before the system crashes.

[0108] In the embodiments of the present application, if the storage system crashes and needs to be reconstructed, the most important thing is the data recovery of the system metadata. Since the real-time snapshot record log records the full state of all system metadata, the data recovery can be performed according to the latest real-time snapshot record log. Further, if there is an incremental operation record log and / or a differential backup record log, it can be considered that the generation time of the incremental operation record log and / or the differential backup record log is later than the generation time of the real-time snapshot record log, that is, the data change of the system metadata occurs on the basis of all the system metadata described in the latest real-time snapshot record log. Therefore, the system reconstruction can be performed by taking the latest real-time snapshot record log as a reference and combining the incremental operation record log and / or the differential backup record log. In this way, the system metadata in the storage system can be restored to the system metadata state before the system crashes, only a limited number of logs after the latest snapshot record log need to be played back, the playback of the full historical record log is avoided, the reconstruction time is greatly shortened, and full data recovery is achieved to the greatest extent.

[0109] In some embodiments, the system log management method provided by the embodiments of the present application can include two parts in whole, the first is the system metadata persistent record storage when the storage system is normally running, and the second is the system metadata recovery after the storage system crashes. The following embodiments are explained and described.

[0110] As Figure 3 shown is a flowchart of the system metadata persistent record storage when the storage system is normally running. After a key-value storage engine is started, its initial metadata is empty. When the first SSTable file is generated (such as MemTable flush), the system generates an incremental operation record log AddFile (0, 100, "a", "z", 1024, 0x12345678). As the operation proceeds, thousands of incremental operation record logs are accumulated in the log. When the number of incremental operation record logs reaches a threshold (such as 10,000), the snapshot process can be triggered, all file information of all levels is serialized, written into a snapshot record log S_1, and all incremental operation record logs before the snapshot record log are deleted.

[0111] In addition, during this period, the system performs a large compression, which combines the 100 files in Level 0 into 10 new files in Level 1. At this time, the system can further generate a differential backup record log according to the 110 incremental operation record logs on the basis of the 100 incremental operation record logs DeleteFile(0,...) and the 10 incremental operation record logs AddFile(1,...), and the content of the differential backup record log can be: UpdateLevelFiles(1, [201, 202,..., 210], ["a", "k",..., "z"], [2048, 4096,...],...), which indicates that the file list of Level 1 is now [201, 202,..., 210] and contains the key range and size of each file and the like.

[0112] As shown in Figure 4 the system metadata recovery after the storage system crashes, if the system crashes at this time, the latest snapshot record log S_n can be found by searching backward from the end of the log, the snapshot record log S_n is loaded, and the system metadata is reconstructed. If a differential backup record log indicating updating the file list of Level 1 is found by scanning forward from S_n, the old list of Level 1 in the memory is directly replaced with the file list indicated by the new differential backup record log, and any incremental operation record log that can exist thereafter is continuously applied, and finally the recovery is completed, and the system metadata in the storage system can be consistent with that before the crash.

[0113] Through the description of the above implementation, those skilled in the art can clearly understand that the method according to the above embodiment can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better implementation.

[0114] As shown in Figure 5 the embodiment of the present application also provides a system log management device, which can include:

[0115] The acquisition module 501 is configured to monitor the data change of the system metadata in the storage system in real time.

[0116] The processing module 502 is configured to, when detecting that the system metadata has data change, generate an incremental operation record log according to the change information of the system metadata, and store the incremental operation record log in the system log.

[0117] The processing module 502 is further configured to generate a differential backup record log according to the plurality of incremental operation record logs in the system log when the differential backup trigger condition is met, store the differential backup record log in the system log, and delete the plurality of incremental operation record logs.

[0118] The processing module 502 is further configured to generate a real-time snapshot record log according to the system metadata in the storage system when the snapshot trigger condition is met, and store the real-time snapshot record log in the system log.

[0119] The processing module 502 is further configured to delete, according to a generation time of the real-time snapshot record log, the incremental operation record logs and the differential backup record logs in the system log whose generation times are earlier than the generation time of the real-time snapshot record log, to update the system log.

[0120] In some embodiments, the processing module 502 is further configured to create an initial snapshot record log when the storage system is initialized, the initial snapshot record log being used to record the system metadata in the storage system at the initialization state.

[0121] The processing module 502 is further configured to store the initial snapshot record log to generate the system log.

[0122] In some embodiments, the processing module 502 is specifically configured to detect a data amount of the data that has been changed when it is detected that the system metadata has undergone a plurality of data changes.

[0123] The processing module 502 is specifically configured to combine the change information of the system metadata corresponding to the plurality of data changes to generate the incremental operation record log if the data amount is less than a preset data amount.

[0124] In some embodiments, the processing module 502 is further configured to determine that the differential backup trigger condition is met when it is detected that the number of the incremental operation record logs in the system log reaches a first number.

[0125] In some embodiments, the processing module 502 is further configured to determine that the differential backup trigger condition is met when it is detected that there is a storage system level change in the plurality of incremental operation record logs stored in the system log.

[0126] In some embodiments, the processing module 502 is further configured to determine that the snapshot trigger condition is met when it is detected that the amount of the record logs stored in the system log reaches a second number, and / or the time interval between the snapshot record logs stored in the system log reaches a preset time length.

[0127] In some embodiments, the processing module 502 is further configured to perform system reconstruction according to the record logs stored in the system log in response to a reconstruction instruction of the storage system, the record logs at least including the real-time snapshot record log.

[0128] In some embodiments, the processing module 502 is specifically used to traverse the system log according to the generation time of each record log stored in the system log, and determine at least one snapshot record log.

[0129] The processing module 502 is specifically used to reconstruct the storage system based on the snapshot log with the smallest difference between the generation time and the current time in at least one snapshot log.

[0130] In some embodiments, the processing module 502 is specifically used to reconstruct the storage system based on the snapshot log with the smallest difference between the generation time and the current time in at least one snapshot log, as well as the incremental operation log and / or differential backup log, if there are still incremental operation logs and / or differential backup logs in the system log.

[0131] In the embodiments of this application, the description of the features corresponding to the system log management device in the embodiments can be found in the relevant description of the system log management method in the embodiments, and will not be repeated here.

[0132] like Figure 6 As shown, embodiments of this application also provide an electronic device, including a memory 601 and a processor 602. The memory 601 stores a computer program, and the processor 602 is configured to run the computer program to perform the steps in any of the above-described system log management method embodiments.

[0133] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described system log management method embodiments at runtime.

[0134] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0135] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described system log management method embodiments.

[0136] The embodiment of the present application further provides another computer program product comprising a nonvolatile computer readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the system log management method embodiments.

[0137] Those skilled in the art will further appreciate that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, various examples have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation should not be interpreted as a departure from the scope of the present application.

[0138] The above describes in detail the process monitoring of the storage system provided by the present application. The principles and implementation manners of the present application are described by using specific examples in the present application. The above description of the examples is only applicable to help understand the method of the present application and its core idea. It should be pointed out that, for those skilled in the art, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A system log management method, characterized in that, The method includes: Real-time monitoring of changes in system metadata within the storage system; When a data change is detected in the system metadata, an incremental operation log is generated based on the change information of the system metadata, and the incremental operation log is stored in the system log. When the differential backup trigger condition is met, a differential backup log is generated based on multiple incremental operation logs in the system log, the differential backup log is stored in the system log, and the multiple incremental operation logs are deleted. When the snapshot triggering conditions are met, a real-time snapshot log is generated based on the system metadata in the storage system, and the real-time snapshot log is stored in the system log. Based on the generation time of the real-time snapshot log, delete the incremental operation log and differential backup log in the system log whose generation time is earlier than the generation time of the real-time snapshot log, so as to update the system log.

2. The method according to claim 1, characterized in that, Before the real-time monitoring of system metadata changes in the storage system, the method further includes: When the storage system is initialized, an initial snapshot log is created. The initial snapshot log is used to record the system metadata in the storage system during the initialization state. The initial snapshot record log is stored to generate the system log.

3. The method according to claim 1, characterized in that, When a data change is detected in the system metadata, an incremental operation log is generated based on the change information in the system metadata, including: When multiple data changes are detected in the system metadata, the amount of data with changes is detected. If the data volume is less than the preset data volume, the change information of the system metadata corresponding to the multiple data changes will be combined to generate the incremental operation log.

4. The method according to claim 1, characterized in that, Before generating a differential backup record log based on multiple incremental operation record logs in the system log when the differential backup trigger condition is met, the method further includes: When the number of incremental operation logs in the system log reaches a first number, it is determined that the differential backup trigger condition is met.

5. The method according to claim 1, characterized in that, Before generating a differential backup record log based on multiple incremental operation record logs in the system log when the differential backup trigger condition is met, the method further includes: When a storage system level change is detected in multiple incremental operation logs stored in the system log, it is determined that the differential backup trigger condition is met.

6. The method according to claim 1, characterized in that, Before generating a real-time snapshot log based on the system metadata in the storage system when the snapshot triggering condition is met, the method further includes: When the number of recorded logs stored in the system log reaches a second quantity, and / or the time interval between the recorded logs and the snapshot logs stored in the system log reaches a preset duration, it is determined that the snapshot triggering condition is met.

7. The method according to claim 1, characterized in that, After updating the system log by deleting incremental operation logs and differential backup logs whose generation time is earlier than the generation time of the real-time snapshot log, the method further includes: In response to the reconstruction command of the storage system, the system is reconstructed according to the record logs stored in the system log, the record logs including at least the real-time snapshot record logs.

8. The method according to claim 7, characterized in that, The system reconstruction in response to the reconstruction command of the storage system, based on the record logs stored in the system log, includes: Based on the generation time of each record log stored in the system log, traverse the system log to determine at least one snapshot record log; The storage system is reconstructed based on the snapshot log with the smallest difference between the generation time and the current time from the at least one snapshot log.

9. The method according to claim 8, characterized in that, The step of reconstructing the storage system based on the snapshot record log with the smallest difference between the generation time and the current time from the at least one snapshot record log includes: If the incremental operation log and / or the differential backup log also exist in the system log, then the storage system is reconstructed based on the snapshot log with the smallest difference between the generation time and the current time in the at least one snapshot log, as well as the incremental operation log and / or the differential backup log.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the system log management method as described in any one of claims 1 to 9 when executing the computer program.

Images