A method, system, device and storage medium for log storage

Abstract

The application provides a log storage method, system, device and storage medium. The method comprises the following steps: setting a configuration item in a log to reserve a preset length for a metadata operation log; in response to the metadata operation log being pruned, determining whether the length of the pruned log is greater than the reserved preset length according to the position of the pruned log; in response to the length of the pruned log being less than or equal to the reserved preset length, retaining the metadata operation log and updating it to the memory; and in response to the length of the pruned log being greater than the reserved preset length, pruning the metadata operation log after the reserved preset length. The application increases the length of the reserved metadata operation log, reads the metadata operation log file when the file system is abnormal, and analyzes, locates and solves the problem.

Description

Technical Field

[0001] This invention relates to the field of distributed file systems, and more specifically, to a method, system, device, and storage medium for log storage. Background Technology

[0002] Metadata is data that describes other data, or structured data that provides information about a resource. Metadata describes information resources or data objects, and its purpose is to: identify resources; evaluate resources; track changes in resources during use; achieve simple and efficient management of large amounts of networked data; and enable effective discovery, retrieval, integrated organization, and effective management of information resources. Current distributed file systems use a journal mechanism (a log of metadata operations) to maintain the atomicity of file system operations (the indivisibility of transactions; all operations within a transaction are either executed continuously or none are executed). This involves placing the metadata information of the operated files in the journal. After the journal is written to disk, the metadata information is modified, ensuring the atomicity of metadata operations in the event of a failure. After a failure, log replay can be used to ensure the correctness of the metadata information. Furthermore, in the existing mechanism, useless journal entries are deleted after events in the log segment are written to disk. The reading, writing, and deletion of the journal are determined by several positions (trimming_pos, trimmed_pos, read_pos, write_pos, etc.). However, when the file system malfunctions, due to the low log level, critical information may not be printed out in the log or only partial information may be printed, which greatly affects the efficiency of problem solving and may even make it difficult to locate the problem. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a method, system, computer device, and computer-readable storage medium for log storage. This invention increases the length of the log for reserved metadata operations without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the log file for metadata operations is read out for analysis, location, and resolution of the problem. This improves the maintainability of the product and enhances its stability and competitiveness without affecting the stability and performance of the storage system.

[0004] To achieve the above objectives, one aspect of the present invention provides a log storage method, comprising the following steps: setting a configuration item in the log to reserve a preset length for the log of metadata operations; in response to pruning the log of metadata operations, determining whether the length of the pruned log is greater than the reserved preset length based on the position of the pruned log; in response to the length of the pruned log being less than or equal to the reserved preset length, retaining the log of metadata operations and updating it in memory; and in response to the length of the pruned log being greater than the reserved preset length, retaining the reserved preset length before pruning the log of metadata operations.

[0005] In some implementations, setting configuration items in the log to reserve a preset length for the log of metadata operations includes recording the write address, the pruned address, the pruned address, and the termination address in the log header.

[0006] In some implementations, determining whether the length of the trimmed log is greater than the reserved preset length based on the position of the trimmed log includes comparing whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0007] In some implementations, the step of pruning the logs for metadata operations after retaining the preset length includes pruning the logs between the pruned address and the already pruned address.

[0008] In another aspect, this invention provides a log storage system, comprising: a setting module configured to set configuration items in the log to reserve a preset length for logs of metadata operations; a judging module configured to, in response to pruning the logs of metadata operations, determine whether the length of the pruned logs is greater than the reserved preset length based on the position of the pruned logs; a first execution module configured to, in response to the length of the pruned logs being less than or equal to the reserved preset length, retain the logs of metadata operations and update them in memory; and a second execution module configured to, in response to the length of the pruned logs being greater than the reserved preset length, retain the reserved preset length before pruning the logs of metadata operations.

[0009] In some implementations, the setting module is configured to record the write address, the pruned address, the pruned address, and the termination address in the log header.

[0010] In some implementations, the determination module is configured to: compare whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0011] In some implementations, the second execution module is configured to: trim the logs between the trimmed address and the trimmed address.

[0012] In another aspect of the present invention, a computer device is provided, comprising: at least one processor; and a memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing the steps of the method described above.

[0013] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0014] The present invention has the following beneficial technical effects: by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system, when the file system is abnormal, the metadata operation log file can be read out for analysis, location and resolution of the problem. Without affecting the stability and performance of the storage system, the maintainability of the product is improved and the stability and competitiveness of the product are enhanced. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0016] Figure 1 A schematic diagram illustrating an embodiment of the log storage method provided by the present invention;

[0017] Figure 2 This is a schematic diagram of log storage in the prior art;

[0018] Figure 3 A schematic diagram of log storage provided by the present invention;

[0019] Figure 4 A schematic diagram of an embodiment of the log storage system provided by the present invention;

[0020] Figure 5 A schematic diagram of the hardware structure of an embodiment of the computer device for log storage provided by the present invention;

[0021] Figure 6 This is a schematic diagram of an embodiment of the computer storage medium for log storage provided by the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0023] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0024] In a first aspect, an embodiment of a log storage method is proposed. Figure 1 The diagram shown is a schematic representation of an embodiment of the log storage method provided by the present invention. Figure 1 As shown, the embodiments of the present invention include the following steps:

[0025] S1. Configure the log to reserve a preset length for the log of metadata operations;

[0026] S2. In response to pruning the logs of metadata operations, determine whether the length of the pruned logs is greater than the reserved preset length based on the position of the pruned logs;

[0027] S3. In response to the log length being less than or equal to the reserved preset length, retain the log of metadata operations and update it in memory; and

[0028] S4. In response to the log length being greater than the reserved preset length, the reserved preset length is retained before the log for metadata operations is trimmed.

[0029] In this embodiment of the invention, the journal refers to the log of metadata operations, named mdlog, which is stored in the metadata pool and continuously appended to. If the journal information can be saved, and log events in the journal can be read using journal tools when the file system malfunctions, it can increase the methods for problem localization.

[0030] This invention proposes a journal optimization method for distributed file systems. By increasing the length of the reserved journal log, when an anomaly occurs in the file system, the journal file can be read out using a journal tool for problem analysis, localization, and resolution. This improves the maintainability of the product and enhances its stability and competitiveness without affecting the stability and performance of the storage system.

[0031] Figure 2 This is a schematic diagram of log storage in the prior art, such as... Figure 2As shown, the log header records information such as write_pos (write address), trimd_pos (trimmed address), expire_pos (expire address), and trim_pos (trimmed address). trimd_pos (trimmed address) identifies the log that has been trimmed. In the existing technology, expire_pos = trim_pos, that is, the trimmed address equals the expiration address. The part between the trimmed address and the trimmed address is the part that will be trimmed in the journal. Journal trimming is performed immediately after the log event is updated in the metadata.

[0032] Configure the log settings to reserve a preset length for metadata operation logs. Add a configuration option; the default reserved journal length is 256MB, which can be modified. Since the current file system has multiple journal types, this configuration reserves a certain length only for the mdlog journal, deprecating other journal types.

[0033] In some implementations, setting configuration items in the log to reserve a preset length for the log of metadata operations includes recording the write address, the pruned address, the pruned address, and the termination address in the log header. Figure 3 A schematic diagram of log storage provided by the present invention, such as Figure 3 As shown, the log header records information such as write_pos (write address), trimd_pos (trimmed address), expire_pos (expire address), and trim_pos (trimmed address). In this embodiment of the invention, the part between the trimmed address trim_pos and the trimmed address trimd_pos is the part that will be trimmed into the journal, and the part between the expire address expire_pos and the trimmed address trim_pos is the reserved journal length. After the log event is updated to the metadata, the journal is trimmed after the reserved length exceeds the default 512M (it can also be set to other sizes).

[0034] In response to log operations on metadata, the system trims the journal, determining whether the length of the trimmed journal exceeds the preset length based on its position. Journal trimming involves promptly deleting unnecessary journals. When trimming a journal, if the trimmed journal position is less than the preset journal length, trimming is not performed. If the trimmed journal position is greater than the preset journal length, a fixed length of journal is retained before trimming.

[0035] In some implementations, determining whether the length of the trimmed log is greater than the preset length based on the position of the trimmed log includes: comparing whether the distance between the termination address and the trimmed address is greater than the preset length. This involves reading `expire_pos` (termination address) and `trim_pos` (trimmed address) from the log header, determining the distance between the termination address and the trimmed address, and then determining whether the distance between the termination address and the trimmed address is greater than the preset length.

[0036] If the length of the pruned log is less than or equal to the reserved preset length, the log for metadata operations is retained and updated in memory. If the length of the pruned log is greater than the reserved preset length, the reserved preset length is retained before pruning the log for metadata operations. That is, if the distance between the termination address and the pruning address is less than or equal to the reserved preset length, the log for metadata operations is retained and updated in memory; if the distance between the termination address and the pruning address is greater than the reserved preset length, the reserved preset length is retained before pruning the log for metadata operations. Journal files can be read using journal tools for problem analysis, location, and resolution. Journal tools can read and modify journals; the read information can be in JSON format or in a line-by-line format.

[0037] In some implementations, the step of pruning the logs for metadata operations after retaining the preset length includes pruning the logs between the pruned address and the already pruned address.

[0038] This invention improves product maintainability and enhances product stability and competitiveness by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the metadata operation log file is read out for problem analysis, location, and resolution.

[0039] It should be noted that the steps in the various embodiments of the above log storage method can be interleaved, substituted, added, or deleted. Therefore, these reasonable permutations and combinations of log storage methods should also fall within the protection scope of this invention, and the protection scope of this invention should not be limited to the embodiments.

[0040] Based on the above objectives, a second aspect of the present invention provides a log storage system. For example... Figure 4As shown, system 200 includes the following modules: a setting module, configured to set configuration items in the log to reserve a preset length for the log of metadata operations; a judgment module, configured to determine whether the length of the truncated log is greater than the reserved preset length based on the position of the truncated log in response to pruning the log of metadata operations; a first execution module, configured to retain the log of metadata operations and update it in memory in response to the length of the truncated log being less than or equal to the reserved preset length; and a second execution module, configured to retain the reserved preset length before pruning the log of metadata operations in response to the length of the truncated log being greater than the reserved preset length.

[0041] In some implementations, the setting module is configured to record the write address, the pruned address, the pruned address, and the termination address in the log header.

[0042] In some implementations, the determination module is configured to: compare whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0043] In some implementations, the second execution module is configured to: trim the logs between the trimmed address and the trimmed address.

[0044] This invention improves product maintainability and enhances product stability and competitiveness by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the metadata operation log file is read out for problem analysis, location, and resolution.

[0045] Based on the above objectives, a third aspect of the present invention provides a computer device, comprising: at least one processor; and a memory storing computer instructions executable by the processor to perform the following steps: S1, setting a configuration item in a log to reserve a preset length for a log of metadata operations; S2, in response to pruning the log of metadata operations, determining whether the length of the pruned log is greater than the reserved preset length based on the position of the pruned log; S3, in response to the length of the pruned log being less than or equal to the reserved preset length, retaining the log of metadata operations and updating it in memory; and S4, in response to the length of the pruned log being greater than the reserved preset length, retaining the reserved preset length before pruning the log of metadata operations.

[0046] In some implementations, setting configuration items in the log to reserve a preset length for the log of metadata operations includes recording the write address, the pruned address, the pruned address, and the termination address in the log header.

[0047] In some implementations, determining whether the length of the trimmed log is greater than the reserved preset length based on the position of the trimmed log includes comparing whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0048] The `expire_pos` (termination address) and `trim_pos` (trimming address) in the log header are read to determine the distance between the termination address and the trimming address. It is then determined whether this distance is greater than a pre-defined length. If the distance is less than or equal to the pre-defined length, the log of the metadata operation is retained and updated in memory. If the distance is greater than the pre-defined length, the pre-defined length is retained before trimming the log of the metadata operation. Journal files can be read using journal tools for problem analysis, location, and resolution. Journal tools can read and modify journals; the retrieved information can be in JSON format or in a line-by-line format.

[0049] In some implementations, the step of pruning the logs for metadata operations after retaining the preset length includes pruning the logs between the pruned address and the already pruned address.

[0050] This invention improves product maintainability and enhances product stability and competitiveness by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the metadata operation log file is read out for problem analysis, location, and resolution.

[0051] like Figure 5 The diagram shown is a hardware structure schematic of an embodiment of the computer device for log storage provided by the present invention.

[0052] For example Figure 5 Taking the device shown as an example, the device includes a processor 301 and a memory 302.

[0053] Processor 301 and memory 302 can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0054] The memory 302, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the log storage method in this embodiment. The processor 301 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 302, thereby implementing the log storage method.

[0055] Memory 302 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by usage according to the log storage method, etc. Furthermore, memory 302 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 302 may optionally include memory remotely located relative to processor 301, and these remote memories can be connected to the local module via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0056] One or more log storage methods are stored in memory 302. When executed by processor 301, the log storage method in any of the above method embodiments is executed.

[0057] The log storage method includes the following steps: setting a configuration item in the log to reserve a preset length for the log of metadata operations; in response to pruning the log of metadata operations, determining whether the length of the pruned log is greater than the reserved preset length based on the position of the pruned log; in response to the length of the pruned log being less than or equal to the reserved preset length, retaining the log of metadata operations and updating it in memory; and in response to the length of the pruned log being greater than the reserved preset length, retaining the reserved preset length before pruning the log of metadata operations.

[0058] In some implementations, setting configuration items in the log to reserve a preset length for the log of metadata operations includes recording the write address, the pruned address, the pruned address, and the termination address in the log header.

[0059] In some implementations, determining whether the length of the trimmed log is greater than the reserved preset length based on the position of the trimmed log includes comparing whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0060] In some implementations, the step of pruning the logs for metadata operations after retaining the preset length includes pruning the logs between the pruned address and the already pruned address.

[0061] The `expire_pos` (termination address) and `trim_pos` (trimming address) in the log header are read to determine the distance between the termination address and the trimming address. It is then determined whether this distance is greater than a pre-defined length. If the distance is less than or equal to the pre-defined length, the log of the metadata operation is retained and updated in memory. If the distance is greater than the pre-defined length, the pre-defined length is retained before trimming the log of the metadata operation. Journal files can be read using journal tools for problem analysis, location, and resolution. Journal tools can read and modify journals; the retrieved information can be in JSON format or in a line-by-line format.

[0062] This invention improves product maintainability and enhances product stability and competitiveness by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the metadata operation log file is read out for problem analysis, location, and resolution.

[0063] Any embodiment of the computer device that performs the above-described log storage method can achieve the same or similar effects as any of the aforementioned method embodiments.

[0064] The present invention also provides a computer-readable storage medium storing a computer program for a method of storing an execution log when executed by a processor.

[0065] like Figure 6 The diagram shown is a schematic representation of an embodiment of the computer storage medium for log storage provided by the present invention. Figure 6 Taking the computer storage medium shown as an example, the computer-readable storage medium 401 stores a computer program 402 that, when executed by a processor, performs the above method.

[0066] The log storage method includes the following steps: setting a configuration item in the log to reserve a preset length for the log of metadata operations; in response to pruning the log of metadata operations, determining whether the length of the pruned log is greater than the reserved preset length based on the position of the pruned log; in response to the length of the pruned log being less than or equal to the reserved preset length, retaining the log of metadata operations and updating it in memory; and in response to the length of the pruned log being greater than the reserved preset length, retaining the reserved preset length before pruning the log of metadata operations.

[0067] In some implementations, setting configuration items in the log to reserve a preset length for the log of metadata operations includes recording the write address, the pruned address, the pruned address, and the termination address in the log header.

[0068] In some implementations, determining whether the length of the trimmed log is greater than the reserved preset length based on the position of the trimmed log includes comparing whether the distance between the termination address and the trimming address is greater than the reserved preset length.

[0069] In some implementations, the step of pruning the logs for metadata operations after retaining the preset length includes pruning the logs between the pruned address and the already pruned address.

[0070] The `expire_pos` (termination address) and `trim_pos` (trimming address) in the log header are read to determine the distance between the termination address and the trimming address. It is then determined whether this distance is greater than a pre-defined length. If the distance is less than or equal to the pre-defined length, the log of the metadata operation is retained and updated in memory. If the distance is greater than the pre-defined length, the pre-defined length is retained before trimming the log of the metadata operation. Journal files can be read using journal tools for problem analysis, location, and resolution. Journal tools can read and modify journals; the retrieved information can be in JSON format or in a line-by-line format.

[0071] This invention improves product maintainability and enhances product stability and competitiveness by increasing the length of the reserved metadata operation log without affecting the current journal mechanism of the file system. When the file system encounters an anomaly, the metadata operation log file is read out for problem analysis, location, and resolution.

[0072] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for the log storage method can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The above computer program embodiments can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0073] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0074] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0075] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0076] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0077] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for storing logs, characterized in that, Includes the following steps: Configure the logs to reserve a preset length for logs of metadata operations; In response to pruning the logs of metadata operations, the length of the pruned logs is determined based on the position of the pruned logs to see if it exceeds the preset length. In response to the pruning of log length being less than or equal to the reserved preset length, the log of metadata operation is retained and the log of metadata operation is updated in memory; as well as If the length of the log to be trimmed is greater than the preset length, the preset length is retained before trimming the log for metadata operations. The provision of a preset length for reserving logs for metadata operations by setting configuration items in the logs includes: Record the write address, the trimmed address, the trimmed address, and the termination address in the log header; The step of determining whether the length of the trimmed log is greater than the preset length based on the position of the trimmed log includes: Compare whether the distance between the termination address and the trimming address is greater than the reserved preset length.

2. The method according to claim 1, characterized in that, The step of trimming the logs for metadata operations after retaining the preset length includes: The logs between the trimmed address and the already trimmed address are trimmed.

3. A log storage system, characterized in that, include: The settings module configures settings items in the logs to reserve a preset length for logs of metadata operations; The judgment module is configured to prune logs in response to metadata operations, and determines whether the length of the pruned log is greater than the reserved preset length based on the position of the pruned log. The first execution module is configured to respond to the pruning of logs with a length less than or equal to the reserved preset length, retain the logs of metadata operations, and update the logs of metadata operations into memory; as well as The second execution module is configured to handle logs that are longer than the reserved preset length, and to retain the reserved preset length before pruning the logs for metadata operations. The settings module is configured to: Record the write address, the trimmed address, the trimmed address, and the termination address in the log header; The judgment module is configured to: Compare whether the distance between the termination address and the trimming address is greater than the reserved preset length.

4. The system according to claim 3, characterized in that, The second execution module is configured to: The logs between the trimmed address and the already trimmed address are trimmed.

5. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-2.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-2.

Images