Log event replaying method and device, electronic equipment and storage medium

Abstract

The invention provides a log event replay method and device, electronic equipment and a storage medium, relates to the technical field of distributed file systems, and is used for replaying a log event through a version conflict pre-judgment and multi-stage write-in coordination mechanism. According to the method, the orderliness of write-in operation, the consistency of data and the integrity of transactions in the log event replay process of the distributed file system can be realized. The method comprises the following steps: acquiring a log event from a distributed file system log locally stored in a client; the log event comprises a metadata path, a local version number and local operation information of the target file; querying a synchronous version number of the target file in the distributed file system from a metadata server based on the metadata path; performing version conflict pre-judgment based on the local version number and the synchronous version number; and if no version conflict is pre-judged, executing a multi-stage write operation to replay the log event.

Description

Technical Field

[0001] This application relates to the field of distributed file system technology, and in particular to a log event replay method, apparatus, electronic device and storage medium. Background Technology

[0002] With the rapid development of cloud computing, big data, and artificial intelligence technologies, reliable storage and efficient access to massive amounts of data have become fundamental to supporting modern information technology applications. Distributed file systems, as a core solution to the problem of massive data storage, provide high scalability, high availability, and high fault tolerance by distributing data across multiple independent servers. In such systems, ensuring data consistency, integrity, and durability after a failure is a key indicator of reliability. Log event replay technology is one of the core mechanisms to achieve this goal; it records and re-executes sequences of data operations, enabling the system to recover from a failure to a consistent state.

[0003] Currently, distributed file systems typically employ an architecture that separates metadata from data. While this architecture improves system scalability, it also introduces new complexities. When the log event replay mechanism is triggered to resume client write operations, it is necessary to coordinate the write order between the metadata layer and the storage layer.

[0004] Traditional log replay methods handle this complex process poorly, resulting in version conflict resolution and data inconsistency; they lack write timing control, which can easily trigger old version operations on the metadata server; and the write order between the storage layer and the metadata layer is disordered, affecting transaction integrity. Summary of the Invention

[0005] This application provides a log event replay method, apparatus, electronic device, and storage medium, which can achieve the orderliness of write operations, data consistency, and transaction integrity during the log event replay process of a distributed file system through version conflict prediction and multi-stage write coordination mechanism.

[0006] Firstly, this application provides a method for replaying log events, including: Log events are retrieved from the distributed file system logs stored locally on the client; the log events include the target file's metadata path, local version number, and local operation information. Based on the metadata path, query the synchronization version number of the target file in the distributed file system from the metadata server; Version conflict prediction is performed based on the local version number and the synchronized version number; If no version conflict is anticipated, a multi-stage write operation is performed to replay the log event; The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to reserve file resources; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on the local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

[0007] The technical solution provided in this application brings at least the following beneficial effects: By comparing the local version number in the log event with the synchronized version number in the metadata server, version conflict prediction is performed before log event replay. Log event replay only occurs when no version conflict is predicted, thus eliminating data inconsistency caused by version conflicts at its source and ensuring data consistency. A multi-stage write sequence of "metadata pre-writing → storage layer data operation → metadata commit writing" establishes clear timing rules for write operations, resolving the problem of missing write timing control and ensuring the orderliness of write operations. Furthermore, file resources are reserved during the metadata pre-writing stage to avoid concurrent conflicts; the global state is updated only after data persistence is ensured during the metadata commit writing stage, mimicking transaction atomicity and resolving the problem of disordered write order, ultimately guaranteeing transaction integrity. Therefore, this application, through version conflict prediction and a multi-stage write coordination mechanism, can achieve the orderliness of write operations, data consistency, and transaction integrity during the replay of log events in a distributed file system.

[0008] One possible implementation method, The version conflict prediction based on the local version number and the synchronized version number includes: Obtain the local latest version number of the target file; If the synchronized version number is greater than or equal to the local version number, or the local latest version number is less than the local version number, a version conflict is predicted, the synchronized version number in the local cache is updated, the replay of the log event is terminated, and the reason for the version conflict is recorded in the system audit log. Otherwise, if no version conflict is anticipated.

[0009] In one possible implementation, the log event further includes a first file creation timestamp and a first last modification time of the target file; before replaying the log event, the method further includes: Query the target file from the metadata server for the second file creation timestamp and the second last modification time; If the second file creation timestamp is different from the first file creation timestamp, or if the second last modification time is greater than the first last modification time, then the replay of the log event is terminated.

[0010] One possible implementation, after obtaining the log event, further includes: Invoke the path resolution function to query the current status of the target file from the metadata server based on the metadata path; If the target file is found to be non-existent, the replay of the log event is terminated, and the error message indicating that the target file is non-existent is recorded in the system audit log. If the query fails due to insufficient permissions, the replay of the log event is skipped, and the permission error information is recorded in the system audit log.

[0011] One possible implementation, after initiating a metadata prewrite request for the target file to the metadata server, further includes: Receive the target indication returned by the metadata server; If the target indication is an operation outdated indication, then the version number of the local cache is updated based on the synchronization version number carried in the operation outdated indication, and the replay of the log event is terminated. If the target indication is a file not found indication, then in the current scenario where the creation log is lost or the target file is a newly created file without metadata, a creation operation is performed so that the metadata server creates metadata for the target file.

[0012] One possible implementation is that the local operation information includes write operations and data to be written; The step of performing corresponding data operations on the target file in the storage layer based on the local operation information includes: Using multi-threaded or asynchronous input / output, the data to be written is divided into blocks and written to the target file of the storage layer, and the data blocks to be written are submitted in sequence. During the writing process, monitor the writing status of each data block of the data to be written; Based on the write status of each data block of the data to be written, identify the data blocks that failed to be written, and rewrite the data blocks that failed to be written into the target file of the storage layer; If the storage layer returns an unrecoverable error indication, the replay of the log event is terminated, and the error type indicated by the error indication and the local version number are recorded.

[0013] One possible implementation, after initiating the metadata commit write request to the metadata server, further includes: If a successful submission indication is received from the metadata server, the replay of the log event is completed. If a version obsolescence error indication is received from the metadata server, the local operations related to the log event are rolled back, the local cached synchronization version number is updated, and the timestamp when the metadata commit write request was initiated, the latest synchronization version number carried by the version obsolescence error indication, and the version obsolescence error indication are recorded in the commit log.

[0014] One possible implementation, prior to performing the multi-stage write operation, further includes: Calculate the time interval between the current time and the last modification time of the target file; If the time interval is less than the minimum write interval, then enter a non-blocking wait state; In the non-blocking waiting state, the waiting time is calculated based on the difference between the minimum write interval and the time interval. If the waiting time is lower than a preset threshold, the waiting time is adjusted to the preset threshold. Release the mutex lock held by the currently threaded thread and wake it up periodically using a condition variable; After each timed wake-up, the synchronization version number of the target file is queried again; If the queried synchronization version number is greater than the local version number, then the replay of the log event is terminated. Otherwise, continue waiting until the accumulated waiting time causes the time interval to reach or exceed the minimum write interval; Record the number of waits, actual wait time, and version changes during the non-blocking wait process to the performance log.

[0015] Secondly, this application provides a log event replay apparatus, comprising: The acquisition module is used to acquire log events from the distributed file system logs stored locally on the client; the log events include the metadata path of the target file, the local version number, and local operation information. The query module is used to query the synchronization version number of the target file in the distributed file system from the metadata server based on the metadata path; The prediction module is used to predict version conflicts based on the local version number and the synchronized version number; The replay module is used to perform a multi-stage write operation to replay the log event if it is predicted that there is no version conflict. The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to reserve file resources; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on the local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

[0016] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the method of the first aspect described above.

[0017] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to implement the method described in the first aspect.

[0018] Fifthly, this application provides a computer program product comprising a computer program; when the computer program is run in an electronic device, the electronic device performs the method described in the first aspect.

[0019] The beneficial effects of the second to fifth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a log event replay method provided in an embodiment of this application; Figure 2 A schematic diagram of the pre-inspection process provided in the embodiments of this application; Figure 3 This is a schematic diagram of the version conflict prediction process provided in the embodiments of this application; Figure 4 This is a schematic diagram of the dual verification process provided in the embodiments of this application; Figure 5 This is a schematic diagram of the metadata pre-writing process provided in an embodiment of this application; Figure 6 This is a schematic diagram of the storage layer data operation process provided in an embodiment of this application; Figure 7 This is a schematic diagram of the metadata submission and writing process provided in the embodiments of this application; Figure 8 This is a schematic diagram of the adaptive write waiting process provided in an embodiment of this application; Figure 9 A schematic diagram of the composition of a log event replay device provided in an embodiment of this application; Figure 10 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The following is a detailed description of a call detail record (CDR) data recording method provided in this application, with reference to the accompanying drawings.

[0022] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0023] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0024] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0025] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0027] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0028] See Figure 1 This is a flowchart illustrating a log event replay method provided in an embodiment of this application. Figure 1 As shown, the log event replay method provided in this application specifically includes the following steps S101~S104.

[0029] S101. Retrieve log events from the distributed file system logs stored locally on the client.

[0030] Specifically, log events are events that occur when operations are performed on a target file, as recorded in the distributed file system log. The client reads the log events to be replayed from the locally stored distributed file system log. These log events include the target file's metadata path, local version number, and local operation information. The target file's metadata path is a unique logical address of the target file within the distributed file system namespace. It does not point to the data itself but is used to query the current state of the target file in the metadata server (MDS). The local version number (le->version) identifies the version of the target file targeted by the historical operations recorded in the log event in local storage. Local operation information includes the operation type (e.g., write operation, create operation, etc.) and the operation content (e.g., data to be written, file creation attributes, etc.).

[0031] In some embodiments, after the log event is obtained, such as Figure 2 As shown, the method also includes: S201. Call the path resolution function to query the current status of the target file from the metadata server based on the metadata path; if the target file does not exist, proceed to S202; if the query fails due to insufficient permissions, proceed to S203. S202. Terminate the replay of log events and record the error message "target file does not exist" to the system audit log; S203. Skip the replay of log events and record permission error information to the system audit log.

[0032] For example, during the pre-check phase, the path resolution function `path_walk()` is called to query the current status of the target file from the metadata server, quickly filtering out invalid operations. `path_walk()` supports symbolic link resolution and path caching, reducing the overhead of repeated queries. For moved or renamed files, the path cache quickly locates the new position. If the target file does not exist (i.e., `path_walk()` returns NULL), meaning the target file has been deleted, the replay of the log event is terminated, and an error message indicating the target file does not exist is recorded in the system audit log. If the query fails due to insufficient permissions (i.e., `path_walk()` returns an `EACCES` error), the replay of the log event is skipped, and a permission error message is recorded in the system audit log.

[0033] In this embodiment, when the target file is detected to be missing or the access permissions are insufficient, the invalid replay operation is no longer performed. Instead, it is immediately terminated or skipped, and the error information is recorded in the system audit log. This can significantly improve the efficiency and reliability of the replay process, while enhancing the observability of the system and providing key evidence for fault diagnosis.

[0034] S102. Based on the metadata path, query the synchronization version number of the target file in the distributed file system from the metadata server.

[0035] Specifically, the synchronization version number sync_ver is maintained by the metadata server (MDS) and represents the global synchronization status of the file in the distributed file system.

[0036] S103. Predict version conflicts based on local version number and synchronized version number.

[0037] In some embodiments, such as Figure 3 As shown, step S103 may include the following sub-steps: S301. Obtain the local latest version number of the target file; S302. Determine whether the synchronous version number is greater than or equal to the local version number, or the local latest version number is less than the local version number; if yes, proceed to S303; if no, proceed to S304. S303. If a version conflict is detected, update the local cached version number, stop replaying log events, and record the reason for the version conflict in the system audit log. S304, no version conflict was predicted.

[0038] For example, the local latest version number ver of the target file is obtained. This local latest version number ver is generated incrementally by client operations and is used to identify the latest modification status of the target file in local storage. For instance, ver is automatically incremented by 1 each time the target file is updated, ensuring the uniqueness of the local operation sequence. If sync_ver ≥ le->version is detected, it means that the log event has been overwritten or synchronized by subsequent operations, and a version conflict is predicted. If ver < le->version is detected, it means that the target file has been modified by a local operation of a higher version, and replaying the current log may cause data rollback, so a version conflict is predicted. If a version conflict is predicted, the locally cached synchronization version number is updated, the replay of log events is terminated, and the reason for the version conflict is recorded in the system audit log.

[0039] In this embodiment, if the synchronized version number of the metadata server is greater than or equal to the local version number in the log event, it indicates that the operation may have been globally overwritten, triggering a version conflict. If the latest local version number currently maintained by the client is less than the local version number in the log event, it indicates that the local operation sequence may be out of order or the state may have rolled back, also triggering a version conflict. Once a version conflict is predicted, the synchronized version number in the local cache is updated, the replay of the log event is immediately terminated, and the cause of the conflict is recorded in the system audit log. This mechanism effectively prevents data inconsistency caused by data overwriting or out-of-order operations, significantly improving the security and reliability of the replay process.

[0040] In some embodiments, the log event further includes a first file creation timestamp and a first last modification time of the target file; before replaying the log event, such as Figure 4 As shown, the method also includes: S401. Query the second file creation timestamp and the second last modification time of the target file from the metadata server; S402. Determine whether the creation timestamp of the second file is different from that of the first file, or whether the last modification time of the second file is greater than that of the first file; if yes, proceed to S403; if no, proceed to S404. S403, Terminate the replay of log events; S404, Continue with the subsequent playback process.

[0041] For example, log events also include the first file creation timestamp le->created_at and the first last modified time le->time of the target file. The second file creation timestamp mds_created_at and the second last modified time mds_mtime of the target file are queried from MDS. If mds_created_at ≠ le->created_at, it means the target file may have been deleted and replaced by a new file with the same name, and replay is immediately terminated. If mds_mtime > le->time, it indicates that MDS holds a newer version, and replay is terminated directly.

[0042] In this embodiment, a dual verification mechanism is introduced before replaying to add a strict security barrier to log event replay: comparing the creation timestamp ensures that the operation object is the same file instance, preventing data corruption of new files with the same name; comparing the last modification timestamp detects whether the target file has been updated by other clients after the log recording, preventing old data from overwriting new data. If either condition is not met, the replay is immediately terminated, thereby fundamentally eliminating the risk of data inconsistency caused by file identity changes or outdated versions before the operation is executed.

[0043] S104. If it is predicted that there is no version conflict, perform a multi-stage write operation to replay the log event.

[0044] The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to notify MDS to reserve file resources, which can ensure that subsequent write operations will not fail due to resource conflicts; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

[0045] In some embodiments, after initiating a metadata pre-write request for the target file to the metadata server, the log event replay method provided in this application further includes: receiving a target indication returned by the metadata server; if the target indication is an operation expiration indication, updating the version number of the local cache based on the synchronization version number carried in the operation expiration indication, and terminating the replay of the log event; if the target indication is a file non-existent indication, performing a creation operation in the case where the current scenario is a log creation loss scenario or the target file is a newly created file without metadata, so that the metadata server creates metadata for the target file.

[0046] For example, such as Figure 5 As shown, the client initiates a metadata prewrite request `mdsclient()->pre_write(in)`, which sends inode information and log version number (i.e., the local version number in the log event) to MDS. MDS processes the request and returns a response. If an operation is outdated (-EOLDOP) is returned, it means that MDS has processed a higher version of the operation, updates the local `sync_ver`, and terminates the replay process. If a file does not exist (-ENOENT) is returned, the cause of the exception is diagnosed. If it is due to a lost creation log, the `create()` operation is automatically triggered to rebuild the metadata; if it is due to a newly created file that has not been created (i.e., `create()` was not executed before prewrite), the `create()` process is completed during the prewrite phase to ensure metadata integrity. Subsequent processes can continue after successful metadata creation. If a success response is returned, subsequent processes can continue.

[0047] In this embodiment, during the metadata pre-writing phase, when a specific error indication is received from the metadata server, the process is not simply terminated. Instead, a differentiated recovery strategy is executed: for an outdated operation indication, the local synchronization version number is immediately synchronized and the system exits safely to prevent data rollback; for a file not found indication, the system intelligently diagnoses whether the root cause is lost creation logs or missing metadata for the file, and automatically triggers a metadata reconstruction process to repair the underlying state. This allows the system to autonomously recover from some critical failures, transforming anomalies that could lead to replay interruptions into a repairable state, ensuring a high success rate for data recovery tasks in complex and abnormal scenarios.

[0048] In some embodiments, local operation information includes write operations and data to be written; based on the local operation information, corresponding data operations are performed on the target file of the storage layer, including: using multi-threaded or asynchronous input / output to write the data to be written in blocks to the target file of the storage layer, and controlling the data blocks to be written to be committed sequentially; monitoring the write status of each data block to be written during the write process; identifying the data blocks that failed to be written based on the write status of each data block to be written, and rewriting the data blocks that failed to be written to the target file of the storage layer; if the storage layer returns an unrecoverable error indication, terminating the replay of the log event, and recording the error type and local version number indicated by the error indication.

[0049] For example, such as Figure 6 As shown, the process begins writing data to the storage layer. `writefile(in, le, ver)` is called to initiate multi-threaded / asynchronous I / O writing. Data is written to the S3 storage layer in blocks, with parallel transmission optimizing performance. The write status of each data block is monitored, and a write status table is maintained. The write process results are judged as follows: If a data block succeeds, all data blocks are committed sequentially to ensure data atomicity and consistency; the storage layer write is successful, and the process proceeds to the next stage. If some data blocks fail, a partial write recovery mechanism is triggered. The system identifies the successfully written data blocks in the status table and re-initiates write requests only for the failed blocks. If an unrecoverable critical error is returned (i.e., `writefile` returns a non-zero error code, such as network timeout or insufficient storage space), the entire replay process is immediately terminated, and the error type and local version number are recorded in the system audit log.

[0050] In this embodiment, multi-threaded or asynchronous I / O is used to write data in parallel blocks to optimize performance, and data atomicity is ensured by final sequential commit. Simultaneously, an innovative write status monitoring and partial recovery mechanism is introduced. When some data blocks fail, it can accurately identify and retransmit only the failed blocks, avoiding overall duplicate writes. For unrecoverable serious errors such as network timeouts and insufficient storage space, the replay process is immediately and safely terminated, and an audit log is recorded. In this way, while maximizing I / O throughput, state-aware differentiated error handling significantly improves the success rate of data persistence and system resource utilization.

[0051] In some embodiments, after initiating a metadata commit write request to the metadata server, the method further includes: if a commit success indication is received from the metadata server, then replaying the log event; if a version outdated error indication is received from the metadata server, then rolling back the local operations related to the log event, updating the synchronization version number of the local cache, and recording the timestamp when the metadata commit write request was initiated, the latest synchronization version number carried by the version outdated error indication, and the version outdated error indication to the commit log.

[0052] For example, such as Figure 7 As shown, the client calls `mdsclient()->write(in, mtime)` to initiate a commit request, sending the updated `mtime` timestamp and `ver` version number to the metadata server. The metadata server verifies whether `ver` and `sync_ver` are consistent. Based on the MDS verification result, a consistency judgment is made: if the verification is consistent, the MDS updates the file metadata (increments `sync_ver`, etc.), the MDS returns a commit success indication to the client, and the client completes the replay of the log event (the process ends successfully); if the verification is inconsistent (`ver` is outdated), the MDS returns `-EOLDOP`, indicating that the committed `ver` version is outdated, the client rolls back the local operations related to the log event, updates the locally cached `sync_ver` (based on the latest version number in the error indication), records the commit log (including the request timestamp, the latest synchronization version number, and the error indication) for troubleshooting, and terminates the replay of the current log event (the process terminates safely).

[0053] This embodiment proposes an eventual consistency guarantee and secure termination mechanism for the metadata commit and write phase. An authoritative version check ensures the atomicity of the transaction: if the check succeeds, the global commit is completed and replayed; if the check fails due to version obsolescence, a secure termination process involving "rolling back local operations, updating cache state, and recording audit logs" is triggered. This not only ensures that inconsistent operations are securely intercepted, preventing data overwriting, but also synchronizes the client state with the metadata server by updating the local cache, and provides complete evidence for fault tracing through detailed logs. Thus, while achieving eventual consistency, it greatly enhances the system's reliability and observability.

[0054] In some embodiments, before performing a multi-stage write operation, such as Figure 8 As shown, the method also includes: S801. Calculate the time interval between the current time and the last modification time of the target file; S802. Determine if the time interval is less than the minimum write interval; if yes, proceed to S803; if no, proceed to S801. S803, Enter non-blocking wait state; S804. In non-blocking wait state, calculate the wait time based on the difference between the minimum write interval and the time interval; S805. Determine whether the waiting time is lower than the preset threshold; if yes, proceed to S806; if no, proceed to S807. S806. Adjust the waiting time to the preset threshold. S807. Release the mutex lock held by the current thread and wake it up periodically using a condition variable; S808. After each timed wake-up, re-query the synchronization version number of the target file; S809. Determine whether the queried synchronization version number is greater than the local version number; if yes, proceed to S810; if no, proceed to S811. S810, Terminate the replay of log events; S811, continue waiting until the accumulated waiting time causes the time interval to reach or exceed the minimum write interval; S812 records the number of waits, actual wait time, and version changes during the non-blocking wait process to the performance log.

[0055] Specifically, a minimum write interval `client_write_replay_interval` is defined to avoid overloading the MDS due to high-frequency replays. For example, setting it to 1 second ensures that each replay interval is at least 1 second. The time interval between the current time and the last modification time of the target file is calculated using `t_interval.to_sec() = enfs_clock_now() - in->mtime`. Here, `t_interval.to_sec()` represents the time interval, `enfs_clock_now()` represents the current time, and `in->mtime` represents the last modification time of the target file. If `t_interval.to_sec() < client_write_replay_interval`, a non-blocking wait state is entered to prevent violations of timing constraints. In the non-blocking wait state, the waiting time `wait_time` is calculated using `wait_time = client_write_replay_interval - t_interval.to_sec()`. If `wait_time < 1 second, for example, `wait_time = 0.5 seconds`, `wait_time` is forcibly set to 1 second to avoid performance degradation caused by frequent wake-ups. During the waiting period, the thread releases the mutex `m_wait_mutex` it holds, allowing other threads to perform path lookups or version checks. It calls the condition variable `m_wait_cond.wait_interval()`, i.e., `m_wait_cond.wait_interval(m_wait_mutex, wait_time)`, to implement timed wake-ups and avoid deadlocks. For example, it automatically checks the file status after waiting for 1 second. After each timed wake-up, during the waiting period, it periodically calls `path_walk()` to reacquire the inode and checks if `sync_ver` and `ver` have changed. If `sync_ver` > `le->version`, it immediately terminates the wait and exits the replay process to prevent writing expired data. Otherwise, it continues waiting until the accumulated `wait_time` makes `t_interval.to_sec()` ≥ `client_write_replay_interval`. It records detailed information on the number of waits, actual wait time, and version changes for performance analysis and optimization.

[0056] In this embodiment, a complete adaptive write waiting mechanism is introduced before performing multi-stage write operations. By dynamically controlling the triggering timing of replay operations, system performance and resource utilization are optimized while ensuring data consistency. First, the time interval since the last modification of the target file is calculated. If this time interval is less than the minimum write interval, a non-blocking waiting state is entered. During the waiting period, the system adaptively adjusts the single wait time (avoiding meaningless short wake-ups) and releases the mutex lock held by the thread to allow other tasks to execute. Then, a condition variable is used to periodically wake the thread. After each wake-up, the system re-queries the synchronization version number of the target file. If the version has been updated (i.e., the file has been modified by another client), replay is immediately terminated to prevent data rollback; otherwise, the waiting continues until the time interval requirement is met. Key metrics throughout the waiting process (such as the number of waits, actual time consumption, and version changes) are recorded in the performance log, providing a basis for system optimization. This achieves intelligent flow control of the metadata server, strict protection of data consistency, and effective utilization of system resources in high-concurrency scenarios.

[0057] In some embodiments, reference counting is used to manage the inode structure in memory. Each time an inode is successfully acquired via `path_walk()`, the reference count is incremented by 1. The reference count is decremented by 1 when the `ll_put()` function is called. When the count reaches 0, the memory occupied by the inode is automatically released. To prevent deadlock, the RAII (Resource Acquisition-Initialization) pattern is used to encapsulate the inode's lifecycle, ensuring that `ll_put()` is called in all branch paths (including error exits) to avoid memory leaks.

[0058] In some embodiments, the system collects the operation time of replaying the entire log event chain, the cumulative number of waits by the adaptive waiting mechanism, and the frequency of version conflicts. It monitors the request response time of the MDS and storage layer (such as S3) to identify potential performance bottlenecks. Based on the real-time system load (such as MDS request latency and CPU utilization), it dynamically adjusts the `client_write_replay_interval` parameter. Under high load, it automatically increases the minimum write interval to protect backend services, and under low load, it decreases the minimum write interval to improve replay speed. Based on access patterns, it optimizes the caching strategy (such as cache size and eviction algorithm) of the `path_walk()` function to reduce cross-node queries and improve path resolution efficiency.

[0059] The technical solutions provided by the above embodiments bring at least the following beneficial effects: The log event replay method provided in this application compares the local version number in the log event with the synchronized version number in the metadata server, performs version conflict prediction before log event replay, and only performs log event replay when no version conflict is predicted. This can fundamentally eliminate data inconsistency problems caused by version conflicts and ensure data consistency. The multi-stage write sequence of "metadata pre-writing → storage layer data operation → metadata commit writing" establishes clear timing rules for write operations, solving the problem of missing write timing control and thus ensuring the orderliness of write operations. Furthermore, reserving file resources in the metadata pre-writing stage avoids concurrent conflicts; updating the global state only after ensuring data persistence in the metadata commit writing stage mimics transaction atomicity, solving the problem of chaotic write order and ultimately ensuring transaction integrity. Therefore, this application can achieve the orderliness of write operations, data consistency, and transaction integrity during the log event replay process of a distributed file system through version conflict prediction and multi-stage write coordination mechanisms.

[0060] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0061] This application embodiment can divide the log event replay device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0062] In some embodiments, this application also provides a log event replay apparatus. The log event replay apparatus may include one or more functional modules for implementing the log event replay method of the above method embodiments.

[0063] For example, Figure 9This is a schematic diagram illustrating the composition of a log event replay device provided in an embodiment of this application. Figure 9 As shown, the log event replay device 900 includes: The acquisition module 901 is used to obtain log events from the distributed file system logs stored locally on the client; the log events include the metadata path of the target file, the local version number, and local operation information; The query module 902 is used to query the synchronization version number of the target file in the distributed file system from the metadata server based on the metadata path; Prediction module 903 is used to predict version conflicts based on local version number and synchronized version number; The replay module 904 is used to perform a multi-stage write operation to replay log events if it is predicted that there is no version conflict. The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to reserve file resources; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

[0064] In some embodiments, the prediction module 903 is specifically used for: Get the latest local version number of the target file; If the synchronized version number is greater than or equal to the local version number, or the local latest version number is less than the local version number, a version conflict is predicted, the synchronized version number in the local cache is updated, the replay of log events is terminated, and the reason for the version conflict is recorded in the system audit log. Otherwise, if no version conflict is anticipated.

[0065] In some embodiments, the log event further includes a first file creation timestamp and a first last modification time of the target file; the device also includes a dual verification module, specifically used for: Before replaying the log events, query the target file's second file creation timestamp and second last modification time from the metadata server; If the creation timestamp of the second file is different from that of the first file, or if the last modification time of the second file is greater than that of the first file, then the replay of the log event will be terminated.

[0066] In some embodiments, the device further includes a pre-check module, specifically used for: After obtaining the log event, the path resolution function is called to query the current status of the target file from the metadata server based on the metadata path; If the target file is found to be non-existent, the replay of the log event is terminated, and an error message indicating that the target file is not found is recorded in the system audit log. If the query fails due to insufficient permissions, the replay of the log event is skipped, and the permission error information is recorded in the system audit log.

[0067] In some embodiments, the replay module 904 is specifically used for: After sending a metadata prewrite request for the target file to the metadata server, receive the target indication returned by the metadata server; If the target indication is an operation outdated indication, the version number of the local cache is updated based on the synchronization version number carried in the operation outdated indication, and the replay of log events is terminated. If the target indicates that the file does not exist, then a creation operation will be performed in the current scenario where the creation log is lost or the target file is a newly created file without metadata, so that the metadata server can create metadata for the target file.

[0068] In some embodiments, the local operation information includes write operations and data to be written; the replay module 904 is specifically used for: Using multi-threaded or asynchronous input / output, the data to be written is written in blocks to the target file of the storage layer, and the data blocks to be written are submitted in sequence. Monitor the write status of each data block to be written during the writing process; Based on the write status of each data block to be written, identify the data blocks that failed to be written, and rewrite the data blocks that failed to be written to the target file of the storage layer; If the storage layer returns an unrecoverable error indication, the replay of the log event is terminated, and the error type and local version number indicated by the error indication are recorded.

[0069] In some embodiments, the replay module 904 is specifically used for: After sending a metadata commit write request to the metadata server, if a commit success indication is received from the metadata server, the replay of the log event is completed. If a version obsolescence error is received from the metadata server, roll back the local operations related to the log event, update the local cached synchronization version number, and record the timestamp when the metadata commit write request was initiated, the latest synchronization version number carried by the version obsolescence error, and the version obsolescence error to the commit log.

[0070] In some embodiments, the apparatus further includes an adaptive write wait module, specifically used for: Before performing a multi-stage write operation, calculate the time interval between the current time and the last modification time of the target file; If the time interval is less than the minimum write interval, it enters a non-blocking wait state; In non-blocking waiting mode, the waiting time is calculated based on the difference between the minimum write interval and the time interval. If the waiting time is lower than the preset threshold, the waiting time is adjusted to the preset threshold. Release the mutex lock held by the currently threaded thread and wake it up periodically using a condition variable; After each scheduled wake-up, the synchronization version number of the target file is queried again; If the queried synchronous version number is greater than the local version number, then the replay of the log event will be terminated. Otherwise, continue waiting until the accumulated waiting time causes the time interval to reach or exceed the minimum write interval; Record the number of waits, actual wait time, and version changes during the non-blocking wait process to the performance log.

[0071] When implementing the functions of the integrated modules described above in hardware, this embodiment of the invention provides a possible schematic diagram of the electronic device involved in the above embodiments. For example... Figure 10 As shown, the electronic device 1000 includes: a processor 1002, a communication interface 1003, and a bus 1004. Optionally, the electronic device 1000 may also include a memory 1001.

[0072] Processor 1002 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1002 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 1002 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0073] Communication interface 1003 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0074] The memory 1001 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0075] As one possible implementation, the memory 1001 can exist independently of the processor 1002. The memory 1001 can be connected to the processor 1002 via the bus 1004 and is used to store instructions or program code. When the processor 1002 calls and executes the instructions or program code stored in the memory 1001, it can implement the log event replay method provided in this embodiment of the invention.

[0076] In another possible implementation, the memory 1001 can also be integrated with the processor 1002.

[0077] Bus 1004 can be an extended industry standard architecture (EISA) bus, etc. Bus 1004 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0078] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0079] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned service invocation device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned service invocation device. Further, the aforementioned computer-readable storage medium can include both internal storage units of the aforementioned service invocation device and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned service invocation device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0080] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to execute any of the log event replay methods provided in the above embodiments.

[0081] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for replaying log events, characterized in that, include: Retrieve log events from the distributed file system logs stored locally on the client; The log events include the target file's metadata path, local version number, and local operation information; Based on the metadata path, query the synchronization version number of the target file in the distributed file system from the metadata server; Version conflict prediction is performed based on the local version number and the synchronized version number; If no version conflict is anticipated, a multi-stage write operation is performed to replay the log event; The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to reserve file resources; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on the local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

2. The log event replay method according to claim 1, characterized in that, The version conflict prediction based on the local version number and the synchronized version number includes: Obtain the local latest version number of the target file; If the synchronized version number is greater than or equal to the local version number, or the local latest version number is less than the local version number, a version conflict is predicted, the synchronized version number in the local cache is updated, the replay of the log event is terminated, and the reason for the version conflict is recorded in the system audit log. Otherwise, if no version conflict is anticipated.

3. The log event replay method according to claim 1 or 2, characterized in that, The log events also include the first file creation timestamp and the first last modification time of the target file; Before replaying the log event, the method further includes: Query the target file from the metadata server for the second file creation timestamp and the second last modification time; If the second file creation timestamp is different from the first file creation timestamp, or if the second last modification time is greater than the first last modification time, then the replay of the log event is terminated.

4. The log event replay method according to claim 1 or 2, characterized in that, After obtaining the log event, the method further includes: Invoke the path resolution function to query the current status of the target file from the metadata server based on the metadata path; If the target file is found to be non-existent, the replay of the log event is terminated, and the error message indicating that the target file is non-existent is recorded in the system audit log. If the query fails due to insufficient permissions, the replay of the log event is skipped, and the permission error information is recorded in the system audit log.

5. The log event replay method according to claim 1 or 2, characterized in that, After initiating a metadata prewrite request for the target file to the metadata server, the method further includes: Receive the target indication returned by the metadata server; If the target indication is an operation outdated indication, then the version number of the local cache is updated based on the synchronization version number carried in the operation outdated indication, and the replay of the log event is terminated. If the target indication is a file not found indication, then in the current scenario where the creation log is lost or the target file is a newly created file without metadata, a creation operation is performed so that the metadata server creates metadata for the target file.

6. The log event replay method according to claim 1 or 2, characterized in that, The local operation information includes write operations and data to be written; The step of performing corresponding data operations on the target file in the storage layer based on the local operation information includes: Using multi-threaded or asynchronous input / output, the data to be written is divided into blocks and written to the target file of the storage layer, and the data blocks to be written are submitted in sequence. During the writing process, monitor the writing status of each data block of the data to be written; Based on the write status of each data block of the data to be written, identify the data blocks that failed to be written, and rewrite the data blocks that failed to be written into the target file of the storage layer; If the storage layer returns an unrecoverable error indication, the replay of the log event is terminated, and the error type indicated by the error indication and the local version number are recorded.

7. The log event replay method according to claim 1 or 2, characterized in that, After initiating the metadata commit write request to the metadata server, the method further includes: If a successful submission indication is received from the metadata server, the replay of the log event is completed. If a version obsolescence error indication is received from the metadata server, the local operations related to the log event are rolled back, the local cached synchronization version number is updated, and the timestamp when the metadata commit write request was initiated, the latest synchronization version number carried by the version obsolescence error indication, and the version obsolescence error indication are recorded in the commit log.

8. The log event replay method according to claim 1 or 2, characterized in that, Before performing the multi-stage write operation, the method further includes: Calculate the time interval between the current time and the last modification time of the target file; If the time interval is less than the minimum write interval, then enter a non-blocking wait state; In the non-blocking waiting state, the waiting time is calculated based on the difference between the minimum write interval and the time interval. If the waiting time is lower than a preset threshold, the waiting time is adjusted to the preset threshold. Release the mutex lock held by the currently threaded thread and wake it up periodically using a condition variable; After each timed wake-up, the synchronization version number of the target file is queried again; If the queried synchronization version number is greater than the local version number, then the replay of the log event is terminated. Otherwise, continue waiting until the accumulated waiting time causes the time interval to reach or exceed the minimum write interval; Record the number of waits, actual wait time, and version changes during the non-blocking wait process to the performance log.

9. A log event replay device, characterized in that, include: The acquisition module is used to retrieve log events from the distributed file system logs stored locally on the client. The log events include the target file's metadata path, local version number, and local operation information; The query module is used to query the synchronization version number of the target file in the distributed file system from the metadata server based on the metadata path; The prediction module is used to predict version conflicts based on the local version number and the synchronized version number; The replay module is used to perform a multi-stage write operation to replay the log event if it is predicted that there is no version conflict. The multi-stage write operation includes: initiating a metadata pre-write request for the target file to the metadata server to reserve file resources; after receiving a successful response from the metadata server, performing corresponding data operations on the target file in the storage layer based on the local operation information; and initiating a metadata commit write request to the metadata server to increment the synchronization version number and update the corresponding metadata.

10. An electronic device, characterized in that, The device includes a processor and a memory, the processor being coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computer device to implement the log event replay method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer-executable instructions that, when executed on a computer, cause the computer to perform the log event replay method according to any one of claims 1 to 8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when run on an electronic device, causes the electronic device to perform the log event replay method as described in any one of claims 1 to 8.

Images