Log replay method and device

Abstract

The embodiment of the present application provides a log replay method and device, the method comprising: determining whether the serial number in the newly added log record and the serial number of the residual log record are continuous; if they are continuous, replaying the log records in sequence according to the current serial number, and ending the log record replay when the serial number is interrupted; if they are not continuous, replaying the log records in sequence according to the current serial number, and ending the log record replay when the log record with the serial number equal to the end serial number is completed. Even if the serial number of the newly added log record and the serial number of the residual log record happen to be connected, the log replay method of the embodiment of the present application will only replay the newly added log record, and will not replay the residual log record, thereby avoiding file system confusion or file anomalies.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a log replay method and device. Background Art

[0002] The file system is the software responsible for managing and storing file information within the operating system. Modifying the file system requires numerous operations, which are not atomic (uninterruptible). If an operation is interrupted, it is likely to cause inconsistent file system data.

[0003] To avoid these problems, journaling file systems were developed. A journaling file system is a dedicated area within the file system that records changes to the file system in advance. If a file system modification is interrupted, restoring the file system simply requires reading the changes recorded in the journaling file system and re-executing the interrupted operation to restore consistency.

[0004] However, in current log file systems, changes to the file system are recorded as transactions in the file system's log area. Each transaction has a unique serial number, and during file system recovery, the transactions are replayed sequentially according to their serial numbers. If a newly generated transaction happens to connect to a transaction at the original location in the partition, the file system recovery process will replay both the newly generated transaction and the original transaction, causing file system corruption or file anomalies. Summary of the Invention

[0005] In order to solve the problem that traditional file system recovery methods replay newly generated transactions together with transactions at the original location of the partition, resulting in file system confusion or file anomalies, the present application provides a log replay method and device.

[0006] In a first aspect, an embodiment of the present application provides a log replay method, the method being applied to a log area, the log area including newly added log records and residual log records, the method comprising:

[0007] When the sequence number of the newly added log record and the sequence number of the remaining log record are not consecutive, and the sequence number of the newly added log record is before the sequence number of the remaining log record, replay the log records in sequence according to the current sequence number, and end the log record replay when the sequence number is interrupted;

[0008] When the sequence number of the newly added log record and the sequence number of the residual log record are consecutive sequence numbers, and the sequence number of the newly added log record is before the sequence number of the residual log record, obtain the end sequence number of the log record that needs to be replayed, and replay the log records in sequence according to the current sequence number, and end the log record replay when the log record with the sequence number of the end sequence number completes the replay.

[0009] In a second aspect, an embodiment of the present application provides a log replay device, the device comprising:

[0010] a determination module configured to determine whether the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers;

[0011] The replay module is configured to:

[0012] When the sequence number of the newly added log record and the sequence number of the remaining log record are not consecutive, replaying the newly added log record;

[0013] When the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers, the end sequence number of the log record to be replayed is obtained, and the newly added log record is replayed until the newly added log record with the end sequence number.

[0014] The technical solution provided by this application has the following beneficial effects: by setting the ending sequence number of the log records to be replayed, when the sequence numbers of the newly added log records and the sequence numbers of the remaining log records are consecutive, only the log records up to the ending sequence number are replayed. Even if the sequence numbers of the newly added log records and the sequence numbers of the remaining log records happen to be connected, the file system recovery process will only replay the newly added log records, not the remaining log records, thereby avoiding file system confusion or file anomalies. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flow chart of a log replay method provided in an embodiment of the present application is shown;

[0017] Figure 2 A schematic diagram showing the distribution of log records on a disk provided by an embodiment of the present application is shown;

[0018] Figure 3 A schematic diagram showing the distribution of another log record on a disk provided by an embodiment of the present application is shown;

[0019] Figure 4 A framework diagram of a log replay device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0021] References throughout this specification to "embodiments," "some embodiments," "one embodiment," or "an embodiment," etc., mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, phrases such as "in embodiments," "in some embodiments," "in at least another embodiment," or "in an embodiment" appearing throughout this specification do not necessarily all refer to the same embodiment. Furthermore, in one or more embodiments, particular features, structures, or characteristics may be combined in any suitable manner. Thus, without limitation, particular features, structures, or characteristics shown or described in connection with one embodiment may be combined in whole or in part with features, structures, or characteristics of one or more other embodiments. Such modifications and variations are intended to be within the scope of this application.

[0022] A file system is a software framework within an operating system that manages and stores file information. It consists of three components: the file system interface, the software suite for object manipulation and management, and objects and attributes. From a system perspective, a file system organizes and allocates space on file storage devices, manages file storage, and provides protection and retrieval of stored files.

[0023] A journaling file system builds upon a traditional file system by adding a log of file system changes. Its design philosophy is to track file system changes and record them in a log. A journaling file system stores log records in a disk partition. If a write operation is interrupted for some reason (such as a power outage), the system will resume the write operation before the interruption based on the log records when it restarts.

[0024] This recovery is atomic because there are only a few cases for an interrupted write operation: no need to re-execute - the write operation is marked as completed; successful re-execution - the write operation is re-executed according to the log; unable to re-execute - the write operation is undone; the log itself is incomplete - the write operation has not been fully written to the log and is therefore simply ignored.

[0025] The EXT4 file system (Fourth Extended File System) is a journaling file system for Linux systems and is the successor to the EXT3 file system. The EXT4 file system typically uses the JDB2 (Journaling Block Device 2) journaling system.

[0026] In the JDB2 logging system, changes to the file system are typically first recorded as transactions in the file system's log area. The operating system then writes these changes to disk. Once the changes are written to disk, the corresponding transactions are discarded (but not erased).

[0027] In the JDB2 logging system, each transaction has a unique sequence number that increases with file system changes. These sequence numbers are continuous. When the system is restarted after a power outage or other reasons, and a partition is remounted, the changes made in the transaction are replayed sequentially based on the sequence number. This process is called file system recovery.

[0028] However, during the recovery process, transactions are replayed starting with sequence numbers. After multiple mounts, the sequence numbers of newly written log records may become consecutive with the sequence numbers of previously incomplete log records (remaining log records). Upon subsequent mounts, the new and remaining log records are replayed together using the consecutive sequence numbers, causing incorrect file system recovery and ultimately leading to file system corruption or file anomalies.

[0029] In order to solve the above problems, the present application provides a log replay method, which sets the transaction end sequence number. When all new log records are replayed, the log replay process ends, avoiding the replay of residual log records and new log records together, making the file system consistent and the files correct.

[0030] like Figure 1 A flow chart of a log replay method is shown, the method comprising the following steps:

[0031] Step S101, when the sequence number of the newly added log record and the sequence number of the residual log record are not consecutive, and the sequence number of the newly added log record is before the sequence number of the residual log record, the log records are replayed in sequence according to the current sequence number, and the log record replay is terminated when the sequence number is interrupted.

[0032] For example, Figure 2As shown, the log storage area of ​​the normal partition includes transactions 1 through 7. After using the optimized write command to write these log records to the new partition in mirrored form, transactions 1 through 4, which contain zero data in the partition, are ignored (this speeds up the write process). The log records remaining in the new partition are transactions 5 through 7. After the mount is started normally, a certain amount of new log records are generated and stored in transactions 1 through 4.

[0033] When the system is mounted again, if new log records are saved in transactions 1 to 3, the sequence numbers of the new log records and the remaining log records will not be consecutive. Log records will be replayed sequentially according to the current sequence number. When the replay reaches transaction 3, the log record replay process ends because the sequence number is not consecutive. In this case, the new log records and the remaining log records will not be replayed together.

[0034] Step S102, when the sequence number of the newly added log record and the sequence number of the residual log record are consecutive sequence numbers, and the sequence number of the newly added log record is before the sequence number of the residual log record, obtain the end sequence number of the log record that needs to be replayed, and replay the log records in sequence according to the current sequence number, and end the log record replay when the log record with the sequence number of the end sequence number completes the replay.

[0035] For example, Figure 3 As shown in the figure, when the system is mounted again, if the newly added log records are saved in transactions 1 to 4, the sequence numbers of the newly added log records and the sequence numbers of the remaining log records are consecutive. At this time, the end sequence number of the log records to be replayed is obtained, and the log replay process is completed by replaying to transaction 4.

[0036] If the end sequence number for log replay is not set, after completing the log replay in transaction4, the log records in transactions5 to 7 will be replayed immediately. In fact, the file changes are recorded as of the log record in transaction4. If the log records in transactions5 to 7 are replayed immediately afterwards, for example, deletion or addition operations are performed on the basis of the final completed file, it will cause file system confusion or file anomalies. In the embodiment of the present application, the end sequence number of the log records that need to be replayed is set in advance, and the log replay process is completed when it is replayed to transaction4, avoiding further file changes, thereby maintaining the consistency of the file and file system.

[0037] In some embodiments, while obtaining the end sequence number of the log record to be replayed, the start sequence number of the log record to be replayed is also obtained. The process of replaying the newly added log record is specifically as follows: replaying the newly added log record with the sequence number of the start sequence number, and replaying the newly added log records in sequence according to the sequence number until the replay of the newly added log record with the sequence number of the end sequence number is completed.

[0038] For example, Figure 3 As shown, if the log records in the partition have not been replayed, the starting sequence number of the newly added log records to be replayed is transaction 1, and the ending sequence number is transaction 4. If the log records in the partition have been replayed, but not all log records have been replayed, the sequence number of the transaction that has not been replayed is used as the starting sequence number. For example, if transaction 1 has been replayed, and all log records after transaction 1 have not been replayed, the starting sequence number is 2. The log replay process is from transaction 2 to transaction 4.

[0039] In some embodiments, the start and end sequence numbers of the above embodiments are stored in the superblock of the EXT4 file system. When remounting a partition, the system driver retrieves the start and end sequence numbers from the superblock. The start sequence number is the s_sequence element in the superblock, and the end sequence number is the s_sequence_end element in the superblock.

[0040] The present invention provides a log replay device for executing Figure 1 The corresponding embodiments, such as Figure 4 As shown, the log replay device provided by this application includes:

[0041] The determination module 201 is configured to: determine whether the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers;

[0042] The replay module 202 is used to perform:

[0043] When the sequence number of the newly added log record and the sequence number of the remaining log record are not consecutive, and the sequence number of the newly added log record is before the sequence number of the remaining log record, replay the log records in sequence according to the current sequence number, and end the log record replay when the sequence number is interrupted;

[0044] When the sequence number of the newly added log record and the sequence number of the residual log record are consecutive sequence numbers, and the sequence number of the newly added log record is before the sequence number of the residual log record, obtain the end sequence number of the log record that needs to be replayed, and replay the log records in sequence according to the current sequence number, and end the log record replay when the log record with the sequence number of the end sequence number completes the replay.

[0045] In some embodiments, while obtaining the end sequence number of the log record that needs to be replayed, the start sequence number of the log record that needs to be replayed is also obtained. The steps of replaying the newly added log record are specifically: starting from the log record with the sequence number of the start sequence number, replaying the log records in sequence according to the sequence number until the log record with the end sequence number is replayed.

[0046] In some embodiments, the start sequence number and the end sequence number are both stored in a super block of the log area.

[0047] In some embodiments, the starting sequence number is the s_sequence element in the super block, and the ending sequence number is the s_sequence_end element in the super block.

[0048] In some embodiments, before replaying the newly added log record, the apparatus further includes:

[0049] a residual log record writing module configured to: write the log record to the log area in a mirrored form using an optimized write command to obtain the residual log record, wherein the optimized write command ignores a portion of the log area with zero data;

[0050] The newly added log record writing module is configured to: write the newly added log record into the log area when the mount is started.

[0051] What has been described above includes examples of implementations of the present invention. For the purposes of describing the claimed subject matter, it is certainly not possible to describe every conceivable combination of components or methods, but it will be appreciated that many additional combinations and permutations of the present innovation are possible. Accordingly, the claimed subject matter is intended to encompass all such changes, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, the above description of the illustrated implementations of this application, including those described in the "Abstract," is not intended to enumerate in detail or to limit the disclosed implementations to the precise forms disclosed. While specific implementations and examples are described herein for illustrative purposes, various modifications that are considered to be within the scope of such implementations and examples are possible, as will be appreciated by those skilled in the relevant art.

[0052] Specifically and with respect to the various functions performed by the aforementioned components, devices, circuits, systems, etc., unless otherwise indicated, the terms used to describe such components are intended to correspond to any component that performs the specified function of the component (e.g., functional equivalent), even if not structurally equivalent to the disclosed structure (which performs the functions of the exemplary aspects of the claimed subject matter shown in this application). In this regard, it will also be recognized that the innovation includes systems and computer-readable storage media having computer-executable instructions for performing the acts and / or events of the various methods of the claimed subject matter.

[0053] The above-mentioned systems / circuits / modules have been described with respect to the interactions between several components / blocks. It will be appreciated that such systems / circuits and components / blocks can include those components or specified subcomponents, some of the specified components or subcomponents and / or additional components, and in accordance with the various permutations and combinations described above. Subcomponents can also be implemented as components that are communicatively coupled to other components rather than being included in a parent component (hierarchical). In addition, it should be noted that one or more components can be combined into a single component that provides aggregate functionality or be divided into several separate subcomponents, and any one or more intermediate layers (e.g., management layers) can be provided to communicatively couple to such subcomponents in order to provide comprehensive functionality. Any component described herein can also interact with one or more other components that are not specifically described herein but are known to those skilled in the art.

[0054] Although the numerical ranges and parameters setting forth the broad scope of the invention are approximate, the numerical values ​​set forth in the specific examples are reported as accurately as possible. However, any numerical value inherently contains certain errors that necessarily arise from the standard deviation found in its corresponding test measurements. In addition, all ranges disclosed herein are to be understood to encompass any and all subranges contained therein. For example, a range of "less than or equal to 11" can include any and all subranges between (and including) a minimum value of zero and a maximum value of 11, i.e., any and all subranges having a minimum value equal to or greater than zero and a maximum value equal to or less than 11 (e.g., 1 to 5). In some cases, the numerical values ​​described for the parameters can have negative values.

[0055] In addition, while a particular feature of the innovation may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include," "including," "having," "containing," and variations thereof and other similar terms are used in the detailed description or claims, these terms are intended to be inclusive in a manner similar to the term "comprising" as an open transition term, without excluding any additional or other elements.

[0056] Reference throughout this specification to "an implementation" or "an implementation" means that a particular feature, structure, or characteristic described in connection with that implementation is included in at least one implementation. Thus, the appearances of the phrase "in one implementation" or "in an implementation" in various places throughout this specification are not necessarily all referring to the same implementation. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more implementations.

[0057] Furthermore, references to "items" or "files" throughout this specification mean that specific structures, features, or objects described in connection with an implementation are not necessarily referring to the same object. Furthermore, "files" or "items" can refer to objects of various formats.

[0058] As used in this application, the terms "node," "component," "module," "system," and the like are generally intended to refer to a computer-related entity that is hardware (e.g., circuitry), a combination of hardware and software, or an entity related to an operating machine having one or more specific functionalities. For example, a component can be (but is not limited to) a process, a processor, an object, an executable, a thread of execution, a program, and / or a computer running on a processor (e.g., a digital signal processor). By way of illustration, both an application running on a controller and the controller can be components. One or more components can reside in a process and / or thread of execution, and a component can be localized on one computer and / or distributed between two or more computers. Although separate components are depicted in various implementations, it is to be appreciated that a component can be represented using one or more common components. Furthermore, the design of various implementations can include different component placement, component selection, and the like to achieve optimal performance. Furthermore, a "device" can take the form of specially designed hardware; generalized hardware that is specialized through the execution of software thereon that enables the hardware to perform specific functions; software stored on a computer-readable medium; or a combination thereof.

[0059] Furthermore, the word "example" or "exemplary" is used in this application to mean "serving as an example, instance, or illustration." Any aspect or design described in this application as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, the use of the word "example" or "exemplary" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean inclusive "or" rather than exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied under any of the aforementioned instances. Additionally, the articles "a" and "an," as used in this application and the appended claims, should generally be construed to mean "one or more," unless otherwise specified or clear from the context to be directed to the singular.

Claims

1. A log replay method, characterized in that: The method is used in a log area, where the log area includes newly added log records and residual log records, and includes: When the sequence number of the newly added log record and the sequence number of the remaining log record are not consecutive, and the sequence number of the newly added log record is before the sequence number of the remaining log record, replaying the log records in sequence according to the starting sequence number of the newly added log record, and ending the log record replay when the sequence number is interrupted; When the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers, and the sequence number of the newly added log record is before the sequence number of the remaining log record, obtaining the end sequence number of the log record that needs to be replayed, and replaying the log records in sequence according to the start sequence number of the newly added log record, and ending the log record replay when the log record with the sequence number of the end sequence number is completed; The residual log record is the original log record that is not ignored when the mirror is written to the new partition and is written to the new partition by the mirror; the newly added log record is the log record written to the new partition after the new partition is mounted.

2. The log replay method according to claim 1, characterized in that: While obtaining the end sequence number of the log record that needs to be replayed, the start sequence number of the log record that needs to be replayed is also obtained. The steps of replaying the log record are specifically: starting from the log record with the sequence number of the start sequence number, replaying the log records in sequence according to the sequence number until the replay is completed to the log record with the sequence number of the end sequence number.

3. The log replay method according to claim 2, characterized in that: The start sequence number and the end sequence number are both stored in the super block of the log area.

4. The log replay method according to claim 1, wherein: Before replaying the log record, the method further includes: Writing the log record to the log area in a mirrored form using an optimized write command to obtain the residual log record, wherein the optimized write command ignores a portion of the log area with zero data; When the mount is started, the newly added log record is written into the log area.

5. A log replay device, characterized in that: include: a determination module configured to determine whether the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers; The replay module is configured to: When the sequence number of the newly added log record and the sequence number of the remaining log record are not consecutive, and the sequence number of the newly added log record is before the sequence number of the remaining log record, replaying the log records in sequence according to the starting sequence number of the newly added log record, and ending the log record replay when the sequence number is interrupted; When the sequence number of the newly added log record and the sequence number of the remaining log record are consecutive sequence numbers, and the sequence number of the newly added log record is before the sequence number of the remaining log record, obtaining the end sequence number of the log record that needs to be replayed, and replaying the log records in sequence according to the start sequence number of the newly added log record, and ending the log record replay when the log record with the sequence number of the end sequence number is completed; The residual log record is the original log record that is not ignored when the mirror is written to the new partition and is written to the new partition by the mirror; the newly added log record is the log record written to the new partition after the new partition is mounted.

6. The log replay device according to claim 5, characterized in that: While obtaining the end sequence number of the log record that needs to be replayed, the start sequence number of the log record that needs to be replayed is also obtained. The steps of replaying the log record are specifically: starting from the log record with the sequence number of the start sequence number, replaying the log records in sequence according to the sequence number until the replay is completed to the log record with the sequence number of the end sequence number.

7. The log replay device according to claim 6, characterized in that: The start sequence number and the end sequence number are both stored in the super block of the log area.

8. The log replay device according to claim 5, characterized in that: Before replaying the newly added log record, the device further includes: a residual log record writing module configured to: write the log record to the log area in a mirrored form using an optimized write command to obtain the residual log record, wherein the optimized write command ignores a portion of the log area with zero data; The newly added log record writing module is configured to: write the newly added log record into the log area when the mount is started.

Images