Metadata log replay processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a metadata log replay processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining a metadata log event, performing local verification on the metadata log event, if the metadata log event is a non-local operation, extracting a metadata field, comparing the metadata field with a preset local index node, and performing data processing according to an operation type when the creation time of the metadata field is updated; updating a file version number based on a preset version incremental rule, and generating dimension control information in combination with the creation time; according to the dimension control information, the data processing process is packaged into a transaction of a preset key value database to be executed, and whether the instruction fails or not is detected; and if the execution result and the dimension control information are successfully written, the preset local index node is updated, and if the execution result and the dimension control information are failed, the operation is revoked in a reverse order and recovered to a transaction pre-encapsulation state. The method is used for guaranteeing the atomicity of metadata log replay, avoiding data residues, solving the problem of metadata synchronization inconsistency and improving the reliability of metadata log replay.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method, apparatus, electronic device and storage medium for metadata log replay processing. Background Technology

[0002] Distributed file systems are widely used in large-scale data storage and collaborative processing scenarios. Metadata, as the core information describing file attributes, status, and relationships, directly determines the reliability and stability of system data management. Metadata log replay is a key technology for achieving multi-node metadata synchronization and restoring data status after node failure in a distributed environment. It is mainly used to solve the metadata synchronization challenges in scenarios such as concurrent operations by multiple clients and offline node recovery, providing technical support for the large-scale deployment and efficient operation of distributed file systems.

[0003] Existing metadata log replay technologies typically begin by retrieving metadata log events from a metadata server. After basic validity checks and filtering of the log events, metadata fields are extracted from the logs and compared with locally stored metadata. Then, based on the operation type corresponding to the log, appropriate data processing operations are performed, ultimately updating the local metadata storage node to complete the replay process for a single log entry. For multiple log entries, a direct execution method based on the received order is often used, which only enables basic log synchronization and metadata update functions.

[0004] Existing technologies cannot restore data to its initial state before operation when some instructions fail to execute during the multi-step data processing of logs. This can easily lead to discrepancies between local metadata and the global state of the metadata server, ultimately causing problems such as metadata chaos and data synchronization failure. It is difficult to guarantee the reliability and data consistency of metadata log replay. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for metadata log replay processing, in order to improve the reliability of metadata log replay and the consistency of metadata.

[0006] In a first aspect, embodiments of this application provide a metadata log replay processing method, including:

[0007] Retrieve metadata log events;

[0008] Perform local operation verification on the metadata log event; if the metadata log event is not a local operation, extract the metadata fields from the metadata log event.

[0009] The metadata fields are compared with the creation time corresponding to the preset local index node;

[0010] When the creation time of the metadata field is later than the creation time of the preset local index node, data processing is performed according to the operation type corresponding to the metadata log event;

[0011] The file version number is updated based on a preset version increment rule, and the updated file version number is combined with the creation time to generate dimension control information.

[0012] The data processing procedure is encapsulated into a transaction in a preset key-value database based on the dimension control information.

[0013] Execute all instructions in the transaction of the preset key-value database and check if there are any instructions that failed to execute;

[0014] If all the instructions are executed successfully, the execution result of the data processing and the dimension control information are written into the preset key-value database and the preset local index node is updated.

[0015] If one or more instructions fail to execute during the process, the executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the transaction that encapsulated the data processing process into the preset key-value database.

[0016] In one possible implementation, data processing is performed based on the operation type corresponding to the metadata log event, including:

[0017] When the operation type is a creation operation, the target path is extracted from the metadata field and the existence of the target path is checked. If the target path does not exist, the preset local index node is directly allocated and the metadata is initialized. If the target path exists, it is determined whether the target object corresponding to the target path is a file or a directory. If it is a file, the file link is deleted; if it is a directory, the empty directory is deleted. The preset local index node is allocated and the file metadata of the preset local index node is initialized. When the operation type is a rename operation, the source path and the target path are extracted from the metadata field. The subtree structure of the source path and the target path is traversed, and the write permission of the parent directory corresponding to the target path is verified. After the verification is passed, an atomic rename is performed, and the corresponding path association information in the preset local index node is updated. When the operation type is a delete link operation, the target path is extracted from the metadata field and the associated nodes of the target path are traversed. The dependency relationship of the associated nodes is verified. After the verification is passed, the local file link is deleted, and the corresponding association record in the preset local index node is cleaned up.

[0018] In one possible implementation, before executing all instructions in the transaction of the preset key-value database, the method further includes:

[0019] The consistency between the dimension control information and the version number and creation time carried by the transaction instruction is verified. If the verification result is inconsistent, the instruction execution is determined to have failed and a rollback process is triggered. The executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the data processing process was encapsulated in the transaction of the preset key-value database. If the verification result is consistent, all instructions in the transaction of the preset key-value database are executed in sequence.

[0020] In one possible implementation, after combining the updated file version number with the creation time to generate dimension control information, the method further includes:

[0021] The dimension control information is written to the preset local index node for persistent storage; a heartbeat data packet is sent to the metadata server according to a preset heartbeat interval, the heartbeat data packet containing the dimension control information stored in the preset local index node and the current operation status; the change information corresponding to the global metadata fed back by the metadata server is received, and the status is verified by combining the dimension control information stored locally; the verified change information is synchronized to the preset local index node, and the corresponding metadata fields are updated.

[0022] In one possible implementation, after synchronizing the verified change information to the preset local index node and updating the corresponding metadata fields, the method further includes:

[0023] When performing a data write operation or a link deletion operation, an operation instruction carrying the dimension control information is simultaneously sent to a preset backend storage; the execution feedback from the preset backend storage is received; if the execution feedback indicates execution failure, the operation instruction carrying the dimension control information is resent to the preset backend storage sequentially at preset tiered time intervals, and the number of retries is recorded; when the number of retries reaches a preset retry threshold and the operation still fails, an exception log is generated and the retry operation is terminated, and the preset local index node is restored to its state before the data write operation or link deletion operation was performed based on the dimension control information.

[0024] In one possible implementation, it also includes:

[0025] Record the processing result and corresponding dimension control information for each metadata log event, and generate a log processing status report; synchronize the log processing status report and the metadata of the preset local index node to the metadata server through a heartbeat interaction link; receive global metadata verification information fed back by the metadata server, the verification information including a global dimension control baseline and an unsynchronized log list; verify the consistency between the processing result and the global status based on the verification information, and perform alignment and correction processing on inconsistent content; synchronize the unsynchronized logs and update the preset local index node.

[0026] Secondly, embodiments of this application provide a metadata log replay processing apparatus, comprising:

[0027] The acquisition module is used to acquire metadata log events;

[0028] The verification module is used to perform local operation verification on the metadata log event. If the metadata log event is not a local operation, the metadata fields in the metadata log event are extracted.

[0029] The comparison module is used to compare the metadata field with the creation time corresponding to the preset local index node;

[0030] The processing module is used to process data according to the operation type corresponding to the metadata log event when the creation time of the metadata field is later than the creation time of the preset local index node;

[0031] The generation module is used to update the file version number based on a preset version increment rule, and combine the updated file version number with the creation time to generate dimension control information.

[0032] An encapsulation module is used to encapsulate the data processing process into a transaction in a preset key-value database based on the dimension control information.

[0033] The execution module is used to execute all instructions in the transaction of the preset key-value database and detect whether there are any instructions that failed to execute.

[0034] The execution module is further configured to, if all the instructions are executed successfully, write the execution result of the data processing and the dimension control information into a preset key-value database and update the preset local index node;

[0035] The execution module is further configured to, if one or more instructions fail to execute during the execution process, revert the executed operations in reverse order, release the occupied system resources, and restore the data to the state before the transaction that encapsulated the data processing process into the preset key-value database.

[0036] In one possible implementation, the processing module is specifically used for:

[0037] When the operation type is a creation operation, the target path is extracted from the metadata field and the existence of the target path is checked. If the target path does not exist, the preset local index node is directly allocated and the metadata is initialized. If the target path exists, it is determined whether the target object corresponding to the target path is a file or a directory. If it is a file, the file link is deleted; if it is a directory, the empty directory is deleted. The preset local index node is allocated and the file metadata of the preset local index node is initialized. When the operation type is a rename operation, the source path and the target path are extracted from the metadata field. The subtree structure of the source path and the target path is traversed, and the write permission of the parent directory corresponding to the target path is verified. After the verification is passed, an atomic rename is performed, and the corresponding path association information in the preset local index node is updated. When the operation type is a delete link operation, the target path is extracted from the metadata field and the associated nodes of the target path are traversed. The dependency relationship of the associated nodes is verified. After the verification is passed, the local file link is deleted, and the corresponding association record in the preset local index node is cleaned up.

[0038] In one possible implementation, the execution module is further configured to:

[0039] The consistency between the dimension control information and the version number and creation time carried by the transaction instruction is verified. If the verification result is inconsistent, the instruction execution is determined to have failed and a rollback process is triggered. The executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the data processing process was encapsulated in the transaction of the preset key-value database. If the verification result is consistent, all instructions in the transaction of the preset key-value database are executed in sequence.

[0040] In one possible implementation, the generation module is specifically used for:

[0041] The dimension control information is written to the preset local index node for persistent storage; a heartbeat data packet is sent to the metadata server according to a preset heartbeat interval, the heartbeat data packet containing the dimension control information stored in the preset local index node and the current operation status; the change information corresponding to the global metadata fed back by the metadata server is received, and the status is verified by combining the dimension control information stored locally; the verified change information is synchronized to the preset local index node, and the corresponding metadata fields are updated.

[0042] In one possible implementation, the generation module is further configured to:

[0043] When performing a data write operation or a link deletion operation, an operation instruction carrying the dimension control information is simultaneously sent to a preset backend storage; the execution feedback from the preset backend storage is received; if the execution feedback indicates execution failure, the operation instruction carrying the dimension control information is resent to the preset backend storage sequentially at preset tiered time intervals, and the number of retries is recorded; when the number of retries reaches a preset retry threshold and the operation still fails, an exception log is generated and the retry operation is terminated, and the preset local index node is restored to its state before the data write operation or link deletion operation was performed based on the dimension control information.

[0044] In one possible implementation, an alignment module is further included, specifically for:

[0045] Record the processing result and corresponding dimension control information for each metadata log event, and generate a log processing status report; synchronize the log processing status report and the metadata of the preset local index node to the metadata server through a heartbeat interaction link; receive global metadata verification information fed back by the metadata server, the verification information including a global dimension control baseline and an unsynchronized log list; verify the consistency between the processing result and the global status based on the verification information, and perform alignment and correction processing on inconsistent content; synchronize the unsynchronized logs and update the preset local index node.

[0046] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0047] The memory stores computer-executed instructions;

[0048] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0049] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0050] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0051] The metadata log replay processing method, apparatus, electronic device, and storage medium provided in this application obtain metadata log events and perform local operation verification, extract metadata fields corresponding to non-local operations, compare the metadata fields with the creation time of preset local index nodes, process data according to operation type when the creation time of the metadata fields is updated, generate dimension control information, and encapsulate the data processing process into a transaction of a preset key-value database for execution. Combined with the result writing and index node update after successful instruction execution, and the reverse order undo and data recovery mechanism after execution failure, the method achieves the effects of ensuring the atomicity of metadata log replay, avoiding data residue, solving the problem of metadata synchronization inconsistency, and improving the reliability of metadata log replay and metadata consistency. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0053] Figure 1 This is a schematic diagram illustrating an application scenario of the metadata log replay processing method provided in the embodiments of this application.

[0054] Figure 2 A flowchart illustrating the metadata log replay processing method provided in this application embodiment;

[0055] Figure 3 A schematic diagram of the structure of the metadata log replay processing device provided in the embodiments of this application;

[0056] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0057] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0058] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0059] First, let me explain the terms used in this application:

[0060] Metadata log events: These are event records kept by the metadata server that describe metadata operations (such as creating, renaming, and deleting links). They are the objects to be processed during metadata log replay.

[0061] Metadata fields: These are the collection of fields containing core metadata information extracted from metadata log events, used for comparison with preset local index nodes and subsequent data processing.

[0062] Preset local index nodes: These are the carriers used locally on the client to store file metadata and dimension control information. They are used for metadata comparison, updates, and persistent storage, and support local verification and state synchronization for log replay.

[0063] File metadata refers to the characteristic information that describes the attributes, status, and relationships of a file or directory. It does not contain the actual business data of the file and is the core storage content of the preset local index node. It needs to be initialized and updated.

[0064] Dimension control information refers to information generated by combining the file version number updated according to the preset version increment rules with the creation time of the metadata log event, which is used for transaction encapsulation and verification, metadata synchronization and state recovery;

[0065] Transactions in a pre-defined key-value database refer to execution units that encapsulate multi-step data processing procedures, based on a pre-defined key-value database. They guarantee the atomicity of operations and support successful commits and failed rollbacks.

[0066] Preset backend storage: refers to the backend storage carrier used to store the results of metadata operations. It receives operation instructions carrying dimension control information and returns the execution results.

[0067] Metadata server: refers to a server that manages global metadata, issues metadata log events, and provides feedback on global metadata verification and change information, thereby aligning the client's local metadata with the global state.

[0068] In existing technologies, data processing is directly executed after simple verification and comparison of metadata log events. The multi-step processing process is not atomically encapsulated, and there is a lack of a sound rollback mechanism for execution failure. This leads to technical problems such as data residue and inconsistent metadata synchronization during the replay of metadata logs, which in turn cause metadata chaos.

[0069] The metadata log replay processing method provided in this application solves the technical problems of inconsistent metadata synchronization and metadata chaos caused by the lack of atomicity guarantee and failure rollback mechanism in the existing metadata log replay. It acquires metadata log events and performs local operation verification, extracts metadata fields corresponding to non-local operations, compares the metadata fields with the creation time of the preset local index node, processes data according to operation type when the creation time of the metadata fields is updated, updates the file version number based on the preset version increment rule, and generates dimension control information in combination with the creation time. Based on the dimension control information, it encapsulates the data processing process into a transaction of the preset key-value database for execution. If all instructions are executed successfully, the execution result and dimension control information are written and the preset local index node is updated. If there is an execution failure, the executed operations are reversed, system resources are released, and the system is restored to the state before the transaction encapsulation. This method solves the technical problems of inconsistent metadata synchronization and metadata chaos caused by the lack of atomicity guarantee and failure rollback mechanism in the existing metadata log replay.

[0070] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0071] Figure 1 This is a schematic diagram illustrating an application scenario of the metadata log replay processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes: terminal 101 and server 102.

[0072] Terminal 101 is used to obtain metadata log events issued by server 102, perform local operation verification on the metadata log events, extract metadata fields corresponding to non-local operations, compare them with the creation time of preset local index nodes, perform data processing according to operation type, generate dimension control information, encapsulate the data processing process into a transaction of a preset key-value database for execution, write the results after transaction commit and perform rollback recovery after index node update or execution failure, and synchronously feed back the log processing status and local metadata to server 102. Server 102 is used to receive status feedback and metadata synchronization requests from terminal 101, issue metadata log events and global metadata verification information and change information, manage global metadata and unsynchronized log lists, perform global verification of the log processing results of terminal 101, feed back the verification results to achieve alignment between the terminal and global metadata status, and generate and store global dimension control benchmarks.

[0073] Figure 2 This is a flowchart illustrating the metadata log replay processing method provided in this application embodiment. The execution entity in this embodiment can be... Figure 1The server 102 in the illustrated embodiment can also be other computer-related devices, and this embodiment is not particularly limited.

[0074] like Figure 2 As shown, the metadata log replay processing method includes the following steps:

[0075] Step S201: Obtain metadata log events.

[0076] Specifically, a metadata log synchronization request is initiated through a heartbeat interaction link established with the metadata server. The system receives metadata log events from the metadata server and records the log's timing information to ensure the integrity and order of log reception, providing a foundation for subsequent sequential processing. Metadata log events are the carriers of metadata operations recorded by the metadata server. Following a timing-based generation principle, the logs contain core information such as the metadata operation type and associated attributes, and are completed through a data transmission mechanism between distributed nodes.

[0077] Step S202: Perform local operation verification on the metadata log event. If the metadata log event is not a local operation, extract the metadata fields from the metadata log event.

[0078] Specifically, local operation verification is performed on the acquired metadata log events to determine whether the operation corresponding to the log was initiated by a local node, and logs of locally initiated operations are filtered out; for log events of non-local operations, the metadata fields contained therein are extracted, including core content such as creation time, operation type, and path information, for subsequent comparison and data processing.

[0079] The metadata corresponding to local operations has been updated locally, so there is no need to repeat log replay. Verification can distinguish between local and non-local operations, avoiding invalid processing. Metadata fields are the core carriers describing metadata changes. After extraction, they can support subsequent comparison and processing operations. Non-local operation logs that need to be synchronized are filtered out, and invalid logs are removed to reduce redundant processing. At the same time, core metadata fields are extracted to provide data support for subsequent creation time comparison, operation type matching, and data processing.

[0080] Step S203: Compare the metadata fields with the creation time corresponding to the preset local index node.

[0081] Specifically, the creation time stored in the preset local index node is retrieved and compared with the creation time contained in the metadata field extracted from the metadata log event to clarify the temporal relationship between the two and generate a comparison result, which serves as the basis for determining whether to perform subsequent data processing.

[0082] The time sequence can be used to determine the age of metadata. Metadata updated at the creation time corresponds to more recent operations and should be synchronized first. Time comparison can initially detect metadata conflicts, preventing old data from overwriting updated local metadata. By completing the determination of metadata age and initial conflict detection, log events that need further processing can be selected based on the comparison results, reducing the system resource consumption caused by invalid data processing and ensuring the rationality and accuracy of metadata synchronization.

[0083] Step S204: When the creation time of the metadata field is later than the creation time of the preset local index node, data processing is performed according to the operation type corresponding to the metadata log event.

[0084] Specifically, the metadata change logic differs for different operation types, requiring targeted processing to ensure that metadata changes comply with the metadata management specifications of the distributed file system; processing only logs updated with creation time can guarantee the timeliness and accuracy of metadata synchronization.

[0085] By making substantial changes to the metadata corresponding to log events, we can adapt to the metadata management needs of different operation types, transform the operations recorded in the logs into specific changes to local metadata, ensure that the metadata changes are in line with the operation requirements, realize the transformation of log operations into local metadata changes, and at the same time ensure the standardization of metadata changes through targeted processing to avoid metadata anomalies caused by incorrect operation type adaptation.

[0086] Step S205: Update the file version number based on the preset version increment rule, and combine the updated file version number with the creation time to generate dimension control information.

[0087] Specifically, the file version number is updated according to the preset version increment rule to generate the updated file version number; the creation time in the metadata field is retrieved, and the updated file version number is associated and integrated with the creation time to generate dimension control information. At the same time, the dimension control information is written to the preset local index node to complete persistent storage, which serves as the benchmark for subsequent verification and synchronization.

[0088] The version increment rule adopts a monotonically increasing mechanism (such as incrementing by one each time). When generating dimension control information in conjunction with the creation time, a unique identifier is constructed by using a preset combination method, such as concatenating the timestamp as the high byte and the version number as the low byte, or by performing a hash operation on the two. The resulting dimension control information can accurately represent the current state of metadata. By constructing a two-dimensional metadata status identifier system, it is ensured that the metadata status can be accurately identified and verified.

[0089] Step S206: Encapsulate the data processing process into a transaction in a preset key-value database based on the dimension control information.

[0090] Specifically, the generated dimension control information is retrieved and used as a verification benchmark. The entire data processing process in the above steps is incorporated into the transaction of the preset key-value database to complete the transaction encapsulation. This ensures that all steps of data processing are integrated into a unified transaction execution unit, while the associated dimension control information is used for consistency verification during the transaction execution process.

[0091] The transactions in the pre-defined key-value database possess atomicity characteristics, enabling unified execution and rollback of multi-step operations. Encapsulation based on dimensional control information ensures precise matching between the transaction execution process and metadata status, guaranteeing the consistency of transaction execution. By integrating scattered data processing steps into atomic execution units, the atomicity of data processing is guaranteed according to transaction characteristics, avoiding metadata chaos caused by residual operations and improving the reliability of metadata processing.

[0092] Step S207: Execute all instructions in the transaction of the preset key-value database and check if there are any instructions that failed to execute.

[0093] Specifically, all data processing instructions contained in the transaction of the preset key-value database are executed in a preset order. During the execution of the instructions, the execution status of each instruction is monitored in real time. The consistency of the dimension control information with the version number and creation time carried by the transaction instructions is checked synchronously. The execution results are recorded, and it is determined whether there are any instructions that have failed to execute.

[0094] Transaction instructions follow the principle of sequential execution to ensure the logical coherence of metadata changes. Consistency checks ensure that transaction execution matches the current metadata state, avoiding metadata state deviations during execution. Execution status detection provides a basis for judgment for subsequent commits or rollbacks, ensuring the accuracy of metadata processing.

[0095] Step S208: If all instructions are executed successfully, the execution results of data processing and dimension control information are written into the preset key-value database and the preset local index node is updated.

[0096] Specifically, when it is detected that all instructions in the transaction have been executed successfully, the execution results of data processing and dimension control information are persisted to the preset key-value database, and the transaction is committed. At the same time, the corresponding metadata fields in the preset local index node are updated synchronously to ensure that the metadata status of the preset key-value database and the preset local index node are consistent, and the metadata is finally persisted.

[0097] After a transaction is committed, the execution result is persisted to ensure that metadata changes can be stably retained. At the same time, the local index node, as the storage carrier of local metadata, needs to be updated synchronously to support subsequent log comparison and synchronization, so as to achieve consistency alignment between the data and the local metadata.

[0098] By persistently storing metadata changes and simultaneously updating the local metadata baseline, we ensure that the metadata status of the preset key-value database and the local index node is consistent, thus guaranteeing that the local metadata is consistent with the metadata status stored in the database and avoiding data synchronization deviations.

[0099] Step S209: If one or more instructions fail to execute during the process, the executed operations are rolled back in reverse order, the occupied system resources are released, and the data is restored to the state before the transaction that encapsulated the data processing process into the preset key-value database.

[0100] Specifically, when a failed instruction is detected during transaction execution, a rollback process is triggered. The executed operations are rolled back one by one in the reverse order of instruction execution, releasing the system resources occupied during execution. Based on dimensional control information, the metadata and related states are accurately restored to the initial state before transaction encapsulation, ensuring that there are no residual changes to the metadata.

[0101] Transaction rollback, based on the atomicity characteristic, reverses the execution of operations, which can offset some of the impact of the execution. Combined with dimensional control information, it can achieve accurate restoration of metadata state, ensuring that the restored state is consistent with that before the transaction was executed. It eliminates metadata residue caused by the failure of some instruction executions, restores metadata to its initial state, releases invalidally occupied system resources, avoids metadata chaos, ensures the integrity and accuracy of metadata state, and ensures the stability of metadata management, providing a clean initial environment for subsequent re-execution of log processing.

[0102] The metadata log replay processing method provided in this embodiment of the invention acquires metadata log events and completes local operation verification and metadata field extraction. It then filters updated metadata by comparing creation time, performs targeted data processing according to operation type, generates dimension control information, encapsulates the data processing process into a transaction of a preset key-value database, executes transaction instructions, and completes persistent updates or reverse rollbacks based on the execution results. This achieves reliable replay of metadata logs, ensures the atomicity of metadata processing, solves the problems of inconsistent metadata synchronization and metadata chaos caused by partial operation failures, improves the reliability of metadata log replay and metadata consistency, and supports the stable operation of distributed file systems.

[0103] This embodiment provides a detailed description of the data processing process based on the operation type corresponding to the metadata log event in the above embodiments. The specific implementation of this process includes the following steps:

[0104] Step a1: When the operation type is a creation operation, extract the target path from the metadata field and check if the target path exists.

[0105] Specifically, after identifying the operation type corresponding to the metadata log event as a creation operation, the target path information is retrieved from the extracted metadata fields, the path detection process is started, the existence of the target path in the local storage environment is checked, and the path existence detection result is generated as the basis for judging the execution of the subsequent creation operation branch.

[0106] Creation operations must be carried out based on a valid and suitable storage path. The existence of the path is the core prerequisite for determining the execution logic of the creation operation branch. By extracting the path information from the metadata field, it can be ensured that the detection object and the creation operation recorded in the log are accurately matched, thus ensuring the validity of the detection results and reducing path-related operation anomalies.

[0107] Step a2: If the target path does not exist, directly allocate a preset local index node and initialize the metadata.

[0108] Specifically, when the target path does not exist, there is no need to perform conflict cleanup operations. Instead, a preset local index node is directly allocated to the file or directory to be created, and the binding between the index node and the object to be created is completed. At the same time, metadata is initialized, and core metadata information such as identifier, version, and time is entered. The initialized metadata is then stored in the allocated preset local index node, thus completing the basic data landing for the creation operation.

[0109] The default local index node is the carrier of local metadata storage. Creating a new object requires binding a dedicated index node to achieve persistent management of metadata. Metadata initialization can assign basic attribute identifiers to the new object, providing a foundation for subsequent metadata comparison, synchronization and version management. By building a local metadata management foundation for the new object, a data carrier is provided for subsequent steps such as dimension control information generation and transaction encapsulation, avoiding subsequent synchronization and verification failures due to missing metadata or unbound index nodes, and improving the efficiency and standardization of creation operations.

[0110] Step a3: If the target path exists, determine whether the target object corresponding to the target path is a file or a directory. If it is a file, delete the file link; if it is a directory, delete the empty directory, allocate a preset local inode, and initialize the file metadata of the preset local inode.

[0111] Specifically, when the target path exists, the type of the target object under the target path is checked to distinguish between files and directories; for file type objects, file link deletion is performed to remove the original file association binding; for directory type objects, empty directories are deleted to clear path occupancy; after completing conflict clearing, a preset local inode is allocated for the object to be created, file metadata is initialized and stored in the inode to ensure that the metadata and the object to be created are accurately matched.

[0112] When the target path exists, the existing object will cause a storage conflict with the newly created object. The conflict needs to be cleared by deleting the corresponding link or empty directory to avoid metadata association confusion. File metadata initialization needs to be completed by relying on the allocated inode to ensure that the storage and management of metadata has a dedicated carrier, guarantee the uniqueness and standardization of metadata, and avoid metadata confusion caused by conflict residue.

[0113] Step a4: When the operation type is a rename operation, extract the source path and target path from the metadata field, traverse the subtree structure of the source path and target path, and verify the write permissions of the parent directory corresponding to the target path.

[0114] Specifically, after identifying the operation type as a rename operation, the source path (original storage path) and target path (new storage path) are extracted from the metadata field respectively; the subtree structure of the two paths is traversed one by one to check the metadata nodes and dependencies associated with the paths; at the same time, write permission is checked on the parent directory corresponding to the target path to check whether the local node has the permission to perform the rename operation in the parent directory, and the subtree traversal result and permission check result are generated.

[0115] Renaming operations involve path changes. By traversing the subtree structure, the metadata nodes associated with the path can be clearly identified, avoiding the loss of metadata due to missing associations during the change process. Write permissions to the parent directory are a legal prerequisite for renaming operations. Permission verification can avoid system anomalies caused by illegal operations, investigate the associated nodes and permission risks of renaming operations, clarify the feasibility of the operation, and provide a prerequisite for subsequent atomic renaming operations. This avoids renaming failures due to missing associations or insufficient permissions, and improves the reliability of the operation.

[0116] Step a5: After successful verification, perform atomic renaming and update the corresponding path association information in the preset local index node.

[0117] Specifically, when the verification passes, an atomic renaming operation is performed to migrate the object corresponding to the source path to the target path as a whole, ensuring that the renaming operation is indivisible and is either executed completely or not at all; the path association information stored in the preset local index node is retrieved synchronously and updated to the association data corresponding to the target path, thus completing the synchronization of local metadata and the renaming operation.

[0118] Atomic operations ensure consistency in the renaming process, preventing path and metadata mismatches caused by partial execution. The path association information in the preset local index node is the core basis for metadata synchronization and must be updated synchronously with the renaming result to ensure that local metadata is consistent with the actual storage path. This completes the synchronization update of object path changes and local metadata, realizing the substantial implementation of the renaming operation and metadata calibration. It ensures accurate matching between local metadata and storage path, guarantees the atomicity and integrity of the renaming operation, avoids path and metadata disconnection, and improves metadata consistency.

[0119] Step a6: When the operation type is a delete link operation, extract the target path from the metadata field, traverse the associated nodes of the target path, and verify the dependency relationship of the associated nodes.

[0120] Specifically, after identifying the operation type as a delete link operation, the target path is extracted from the metadata field, all associated nodes corresponding to the path are traversed, and the dependencies between nodes are sorted out; the dependencies are verified to check whether there are any unresolved mandatory dependencies (such as other objects associated with the link node), confirming that the deletion operation will not cause the metadata association to break, and generating dependency verification results.

[0121] Deleting links requires avoiding metadata anomalies caused by unresolved dependencies. The dependencies between related nodes directly affect the legitimacy of the deletion operation. By traversing and verifying, the dependency situation can be fully understood, avoiding system failures caused by accidental deletion, effectively identifying potential dependency issues in link deletion operations, confirming the feasibility of the operation, providing a preliminary judgment for subsequent deletion operations, and avoiding metadata association chaos caused by unresolved dependencies.

[0122] Step a7: After successful verification, delete the local file link and clean up the corresponding associated records in the preset local index node.

[0123] Specifically, when the verification is successful, a local file link deletion operation is performed to unbind the link corresponding to the target path; at the same time, the associated record corresponding to the file link in the preset local index node is retrieved to complete the cleanup operation, ensuring that there is no residual invalid associated data in the index node, and realizing synchronous cleanup of metadata.

[0124] After deleting a file link, the associated records in the local metadata must be cleaned up simultaneously to avoid invalid data residues that could cause deviations in subsequent metadata comparisons and synchronization. The local index node is preset as the metadata storage carrier, and its data integrity directly affects the accuracy of metadata log replay, so it must be updated synchronously with the link deletion operation.

[0125] This invention implements targeted processing procedures for three operation types: creating, renaming, and deleting links. These procedures complete path extraction, existence detection, dependency verification, permission checking, and metadata initialization and updating. By combining atomic operations and conflict cleanup mechanisms, the invention achieves standardized and precise metadata processing, ensures consistency between metadata changes and log operations, supports reliable execution of metadata log replay, solves the problems of incompatible metadata processing and chaotic associations under multiple operation types, and improves the stability and accuracy of metadata management.

[0126] This embodiment provides a detailed description of the process prior to executing all instructions in a transaction of the preset key-value database in the above embodiments. The specific implementation of this process also includes the following steps:

[0127] Step b1 verifies the consistency between the dimension control information and the version number and creation time carried by the transaction instruction.

[0128] Specifically, before executing all instructions in a transaction of a preset key-value database, the generated dimension control information is retrieved, and the file version number and creation time contained therein are extracted. At the same time, the version number and creation time carried in the transaction instructions are extracted, and the two sets of version numbers and two sets of creation times are compared to determine whether the data is completely matched. A consistency verification result is generated as the core basis for determining whether the transaction can be executed.

[0129] Dimensional control information is a two-dimensional benchmark representing the current state of metadata. The version number and creation time carried by the transaction instruction must match the benchmark completely to ensure that the transaction execution is based on the latest and valid metadata state, avoid the disconnect between the transaction execution and the current state of metadata, avoid metadata update errors caused by version or time inconsistencies, improve the accuracy of transaction execution, and lay the foundation for the atomic execution of subsequent transactions.

[0130] Step b2: If the verification result is inconsistent, the instruction execution is determined to have failed and a rollback process is triggered. The executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the transaction in the preset key-value database was encapsulated.

[0131] Specifically, when the consistency check result is a mismatch, the transaction instruction execution is directly determined to have failed, and the transaction execution is directly terminated (without executing any in-transaction instructions). The rollback process is only used to clean up temporary resources that may be generated during the verification phase, and to accurately restore all related data to the state before the transaction when the data processing process was encapsulated into the preset key-value database, ensuring that there are no residual changes to the metadata.

[0132] Transactions possess atomicity; inconsistencies in state verification indicate a risk of data deviation during transaction execution, necessitating a rollback mechanism to offset the impact of executed operations. Reverse undoing can accurately restore the initial state of each operation step, and combined with dimensional control information, it can ensure the accuracy of the recovery results and avoid metadata residue anomalies.

[0133] Step b3: If the verification results are consistent, then execute all instructions in the transaction of the preset key-value database in sequence.

[0134] Specifically, when the consistency check result is a match, it is confirmed that the transaction instruction is consistent with the current state of the metadata and is ready for execution. According to the preset instruction execution order in the transaction, all instructions in the transaction of the preset key-value database are executed in sequence. During the execution process, the execution status of each instruction is monitored in real time, and key data in the execution process is recorded synchronously to provide a basis for subsequent result judgment.

[0135] Transaction instructions follow the principle of sequential execution, ensuring the logical consistency of metadata changes and avoiding metadata association errors caused by out-of-order execution of instructions. Based on the premise that consistency verification is passed, it can be guaranteed that the execution of each instruction is consistent with the current state of metadata, reducing instruction execution anomalies.

[0136] This invention achieves precise risk control and anomaly handling before transaction execution by verifying the consistency of dimensional control information with the version number and creation time of transaction instructions. In cases of inconsistent verification, it triggers reverse rollback and restores the initial state; in cases of consistent verification, it executes transaction instructions in sequence. This ensures the atomicity and accuracy of transaction execution, solves the problem of metadata chaos caused by the deviation between transaction execution and metadata state, and improves the reliability of transaction processing and metadata consistency during metadata log replay.

[0137] This embodiment provides a detailed description of the process in the above embodiment where the updated file version number is combined with the creation time to generate dimension control information. The specific implementation of this process also includes the following steps:

[0138] Step c1: Write the dimension control information into a preset local index node for persistent storage.

[0139] Specifically, after generating dimension control information, the information and the associated identifier of the corresponding metadata log event are retrieved, and the dimension control information is written to the specified storage area of ​​the preset local index node to complete the data persistence process, ensuring that the information will not be lost due to system interruption or temporary failure. At the same time, the storage timestamp of the preset local index node is updated to mark the latest update status.

[0140] By retaining a dual-dimensional benchmark of metadata status, stable data support is provided for subsequent transaction execution verification, global status comparison and anomaly recovery. At the same time, the update mark of local metadata is completed to ensure that the information stored in the local index node is up-to-date, thereby improving the stability of local metadata storage.

[0141] Step c2: Send a heartbeat data packet to the metadata server according to the preset heartbeat interval. The heartbeat data packet contains the dimension control information and the current operation status stored in the preset local index node.

[0142] Specifically, according to the preset heartbeat time interval, the heartbeat interaction process is initiated, the latest stored dimension control information is retrieved from the preset local index node, and the current metadata processing operation status (such as log processing progress and transaction execution status) is collected synchronously. The two types of information are packaged into a heartbeat data packet and sent to the metadata server through the inter-node communication link to ensure the integrity of data transmission.

[0143] The heartbeat mechanism is a way for distributed nodes to maintain communication and synchronize their states. By sending data packets at fixed intervals, it ensures that the metadata server has real-time knowledge of the metadata status of the local node, establishes a communication link between the local and global nodes, lays the foundation for receiving global change information and performing state alignment, and confirms the communication connectivity between the local node and the metadata server.

[0144] Step c3: Receive change information corresponding to global metadata from the metadata server and complete status verification by combining it with the dimension control information stored locally.

[0145] Specifically, the system receives global metadata change information from the metadata server via a heartbeat interaction link. This global metadata change information includes updated metadata content and global dimension control benchmarks. The system then retrieves dimension control information stored in the local preset index node, compares the local benchmark with the global benchmark and the local metadata with the global change information one by one, verifies data consistency, generates status verification results, and clarifies the deviation between local and global data.

[0146] By investigating discrepancies between local and global metadata, the changes that need to be synchronized are identified, invalid change information is filtered out, and accurate basis is provided for subsequent local metadata updates. This avoids metadata chaos caused by blind synchronization, accurately identifies inconsistencies between local and global metadata, clarifies the scope of synchronization, provides direction for subsequent synchronization operations, reduces system resource consumption caused by invalid synchronization, and ensures the accuracy of metadata synchronization.

[0147] Step c4: Synchronize the verified change information to the preset local index node and update the corresponding metadata fields.

[0148] Specifically, by filtering out valid global metadata change information and excluding invalid or conflicting changes, valid change information is synchronized to preset local index nodes, and the corresponding metadata fields in the index nodes are updated one by one to ensure that local metadata is consistent with global change information. The dimension control information stored in the preset local index nodes is updated synchronously, the metadata update status is marked, and the local and global metadata are aligned.

[0149] The verified change information is legal and adaptable. Synchronizing it to the local index node can unify local and global metadata. The synchronous update of metadata fields and dimension control information can ensure that the local benchmark is consistent with the global benchmark, realize the consistency and synchronization of local and global metadata, eliminate metadata deviation, ensure the accuracy and timeliness of local metadata, provide reliable local data support for subsequent steps of metadata log replay, and improve the uniformity of metadata management in distributed systems.

[0150] This invention persistently stores dimensional control information in a preset local index node, feeds back the local status to the metadata server at preset heartbeat intervals, receives global metadata change information and completes consistency verification, and updates the local metadata with the verified change information, thereby achieving precise alignment between local and global metadata, ensuring the consistency and timeliness of metadata status, solving the problem of local metadata being out of sync with global status, supporting reliable execution of metadata log replay, and improving the stability and uniformity of distributed file system metadata management.

[0151] This embodiment provides a detailed explanation of the process described in the above embodiment, which involves synchronizing the verified change information to the preset local index node and updating the corresponding metadata fields. The specific implementation of this process includes the following steps:

[0152] Step d1: When performing a data write operation or a link deletion operation, an operation instruction carrying dimension control information is sent synchronously to the preset backend storage.

[0153] Specifically, when a data write operation or a link deletion operation is triggered, the dimension control information stored in the preset local index node is retrieved, associated and encapsulated with the corresponding operation instruction, and then synchronously sent to the preset backend storage via the communication link between the node and the backend storage. This ensures that the instruction and dimension control information are transmitted synchronously and completely. This provides the preset backend storage with an operational basis and metadata status benchmark, enabling synchronous linkage between local and backend storage operations. It ensures that the operations performed by the backend storage are consistent with local metadata changes, avoiding operations that are disconnected from local metadata and reducing metadata inconsistencies caused by backend operation anomalies.

[0154] Step d2: Receive execution feedback from the preset backend storage.

[0155] Specifically, after sending the operation command, the system maintains a smooth communication link with the preset backend storage, waits for and receives the execution feedback returned by the backend storage, parses the feedback content to determine the execution result of the operation command (success or failure), and synchronously records the feedback time and core feedback information. By obtaining the operation execution result from the backend storage, it is clear whether the operation has been successfully implemented, timely grasps the operation implementation status, avoids duplicate operations or metadata residues caused by not being aware of the backend execution result, and ensures the traceability of operation execution.

[0156] Step d3: If the execution feedback indicates failure, the operation instructions carrying dimension control information are resent to the preset backend storage sequentially according to the preset stepped time intervals, and the number of retries is recorded.

[0157] Specifically, when the parsing execution feedback fails, a tiered automatic retry mechanism is triggered. According to the preset tiered time interval (the interval increases with the number of retries), the same operation instruction carrying dimension control information is re-encapsulated and sent to the preset backend storage in sequence. Each time a retry instruction is sent, the number of retries is recorded synchronously, and the current number of retries is compared with the preset retry threshold in real time to determine whether to continue retrying.

[0158] Tiered time intervals can avoid frequent retries in a short period of time that would consume system and backend storage resources. They can also accommodate possible temporary failures in backend storage (such as network fluctuations or resource consumption). Increasing the intervals increases the probability of successful operation execution. Recording the number of retries can prevent infinite retries, improve the success rate of operation execution, control resource consumption through tiered intervals, and manage the retry process by recording the number of retries, thus avoiding system burden caused by invalid retries.

[0159] Step d4: When the number of retries reaches the preset retry threshold and the operation still fails, an exception log is generated and the retry operation is terminated. Based on the dimension control information, the preset local index node is restored to the state before the data writing operation or the link deletion operation was performed.

[0160] Specifically, when the recorded number of retries reaches the preset retry threshold and the backend storage still has not returned a successful execution feedback, the retry operation is terminated, an exception log is generated, and core information such as operation type, number of retries, reason for failure, and execution time are recorded for subsequent troubleshooting. Simultaneously, the metadata state before the operation recorded in the dimension control information is retrieved. Based on this baseline, the metadata of the preset local index node is accurately restored to the state before the data write or link deletion operation, ensuring no residual changes to the metadata. The preset retry threshold avoids unlimited retries consuming resources; the dimension control information records the metadata baseline before the operation, enabling accurate restoration of the metadata state, ensuring that the restored state is consistent with the state before the operation, avoiding metadata chaos caused by backend operation failures, and guaranteeing the integrity and accuracy of local metadata.

[0161] This invention improves the execution and exception handling loop of backend operations by synchronously sending instructions carrying dimension control information, receiving execution feedback, triggering a tiered retry mechanism, terminating the operation and restoring the metadata state after reaching the retry threshold, thereby increasing the success rate of backend operation execution, avoiding metadata change residues, solving the problem of metadata chaos caused by backend operation failures, ensuring the consistency between local metadata and backend storage operations, and supporting the reliable execution of metadata log replay.

[0162] In one specific embodiment, the following steps are also included:

[0163] Step e1: Record the processing result and corresponding dimension control information for each metadata log event, and generate a log processing status report.

[0164] Specifically, for each completed metadata log event, its processing result (success or failure) is recorded in real time. The dimension control information corresponding to the log event is retrieved and the two are associated and bound. All associated data are summarized to generate a log processing status report. The report includes the identifier of each log, the processing result, the corresponding dimension control information, and the processing time, ensuring that the report data is complete and traceable, and connecting to the subsequent global synchronization process.

[0165] By retaining the processing traces and metadata status of each local log, a standardized status report is generated, providing basic data for subsequent verification and alignment with global metadata. This also facilitates subsequent troubleshooting and process review, achieving standardized recording of local log processing status, ensuring the traceability of the log processing process, providing complete and accurate local data support for subsequent global status verification and alignment, and reducing alignment deviations caused by data loss.

[0166] Step e2: Synchronize the log processing status report and the metadata of the preset local index node to the metadata server through the heartbeat interaction link.

[0167] Specifically, the generated log processing status report is retrieved, the latest metadata and dimension control information stored in the preset local index node are obtained synchronously, and the three types of data are packaged and integrated into a synchronous data packet. Relying on the established heartbeat interaction link, the data packet is synchronized to the metadata server according to the preset communication specifications to ensure data transmission integrity and accurate timing, and the transmission status is recorded synchronously to confirm delivery.

[0168] By feeding back the complete picture of local log processing and the latest metadata status to the metadata server, accurate communication between local data and the metadata server is achieved. This ensures that the server has a comprehensive understanding of local log processing and metadata status, laying the foundation for subsequent global verification and deviation correction, and improving the accuracy of global metadata control.

[0169] Step e3: Receive global metadata verification information from the metadata server. The verification information includes the global dimension control baseline and the list of unsynchronized logs.

[0170] Specifically, the system receives global metadata verification information returned by the metadata server through the heartbeat interaction link, parses the core content of the information, extracts the global dimension control benchmark (a unified standard for comparing the local and global metadata status) and the unsynchronized log list (a set of log events that have not been synchronized locally), sorts out the relationship between the two types of information, and provides a clear basis for subsequent consistency verification and log completion.

[0171] By obtaining the comparison benchmark of local and global metadata and the information of unsynchronized logs, the deviation range of local metadata and the content to be supplemented are clarified, providing a precise direction for subsequent consistency verification, alignment correction and log synchronization, and avoiding blind operation; the gap between local and global metadata is accurately located, the tasks to be processed are clarified, and a clear basis is provided for subsequent alignment operations, reducing the system resource consumption caused by invalid verification and synchronization, and improving the efficiency of global alignment.

[0172] Step e4: Verify the consistency between the processing result and the global state based on the verification information, and perform alignment correction processing on the inconsistent content.

[0173] Specifically, based on the global verification information fed back by the metadata server, the local log processing results, the metadata of the preset local index node, and the corresponding dimension control information are compared one by one with the global dimension control benchmark to verify their consistency. For any inconsistencies found in the comparison, the local metadata and dimension control information are corrected according to the global dimension control benchmark, and the associated data in the preset local index node is updated to ensure that the local state is aligned with the global standard.

[0174] The global dimension control baseline serves as a unified reference for the metadata of all nodes in a distributed system. Consistency checks and corrections based on this baseline ensure the uniformity of metadata states across all nodes, preventing the amplification of local deviations. By eliminating discrepancies between local log processing results and metadata and the global state, local and global state unification is achieved, avoiding metadata chaos, ensuring the consistency of metadata log replay, eliminating metadata anomalies caused by local deviations, ensuring the uniformity of metadata standards across nodes in the distributed system, and supporting the reliable execution of metadata log replay.

[0175] Step e5: Synchronize the unsynchronized logs and update the preset local index node.

[0176] Specifically, all log events in the unsynchronized log list are retrieved, and the complete process of metadata log replay is followed. Log verification, field extraction, comparison, data processing, transaction encapsulation and execution, and result submission are performed one by one to complete the unsynchronized log completion process. The metadata and corresponding dimension control information in the preset local index node are updated synchronously, the synchronization status of the unsynchronized logs is marked, and the final update of the local metadata is completed.

[0177] Unsynchronized logs are the cause of discrepancies between local and global metadata. Synchronizing logs according to the standard log replay process ensures that metadata changes corresponding to unsynchronized logs are accurately implemented. Synchronously updating local index nodes ensures that local metadata matches the global state in real time, completes unsynchronized local log events, improves the local metadata system, ensures that local metadata covers all global changes, eliminates the gap between local and global metadata caused by unsynchronized logs, ensures that local metadata is completely consistent with the global state, improves the integrity and reliability of metadata log replay, and supports the stable operation of the distributed file system.

[0178] This invention, through recording log processing results and dimension control information to generate a status report, synchronizing it to the metadata server, receiving global verification information, verifying consistency and correcting deviations, synchronizing unsynchronized logs and updating local index nodes, completes global state alignment after metadata log replay, solves the problem of local metadata being disconnected from global state, achieves accurate unification of local and global metadata, improves the integrity and reliability of metadata log replay, and ensures metadata consistency in distributed systems.

[0179] Figure 3 This is a schematic diagram of the metadata log replay processing apparatus provided in an embodiment of this application. Figure 3 As shown, the metadata log replay processing device 30 includes:

[0180] Module 301 is used to retrieve metadata log events;

[0181] The verification module 302 is used to perform local operation verification on the metadata log event. If the metadata log event is not a local operation, the metadata fields in the metadata log event are extracted.

[0182] The comparison module 303 is used to compare the metadata fields with the creation time corresponding to the preset local index node;

[0183] Processing module 304 is used to process data according to the operation type corresponding to the metadata log event when the creation time of the metadata field is later than the creation time of the preset local index node.

[0184] The generation module 305 is used to update the file version number based on a preset version increment rule, and combine the updated file version number with the creation time to generate dimension control information.

[0185] Encapsulation module 306 is used to encapsulate the data processing process into a transaction of a preset key-value database based on dimension control information;

[0186] The execution module 307 is used to execute all instructions in the transaction of the preset key-value database and detect whether there are any instructions that failed to execute.

[0187] The execution module 307 is also used to write the execution results of data processing and dimension control information into the preset key-value database and update the preset local index node if all instructions are executed successfully.

[0188] The execution module 307 is also used to, if one or more instructions fail to execute during the execution process, revert the executed operations in reverse order, release the occupied system resources, and restore the data to the state before the transaction that encapsulated the data processing process into the preset key-value database.

[0189] In one possible implementation, the processing module 304 is specifically used for:

[0190] When the operation type is a creation operation, the target path is extracted from the metadata field and its existence is checked. If the target path does not exist, a preset local index node is directly allocated and its metadata is initialized. If the target path exists, it is determined whether the target object corresponding to the target path is a file or a directory. If it is a file, the file link is deleted; if it is a directory, the empty directory is deleted. A preset local index node is allocated and its file metadata is initialized. When the operation type is a rename operation, the source path and target path are extracted from the metadata field. The subtree structure of the source path and target path is traversed, and the write permissions of the parent directory corresponding to the target path are verified. After verification, an atomic rename is performed, and the corresponding path association information in the preset local index node is updated. When the operation type is a delete link operation, the target path is extracted from the metadata field, and the associated nodes of the target path are traversed. The dependencies of the associated nodes are verified. After verification, the local file link is deleted, and the corresponding associated records in the preset local index node are cleaned up.

[0191] In one possible implementation, the execution module 307 is further configured to:

[0192] The consistency between the dimension control information and the version number and creation time carried by the transaction instruction is verified. If the verification result is inconsistent, the instruction execution is determined to have failed and a rollback process is triggered. The executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the data processing process was encapsulated in the transaction of the preset key-value database. If the verification result is consistent, all instructions in the transaction of the preset key-value database are executed in sequence.

[0193] In one possible implementation, the generation module 305 is specifically used for:

[0194] The dimension control information is written to a preset local index node for persistent storage; heartbeat data packets are sent to the metadata server at preset heartbeat intervals, and the heartbeat data packets contain the dimension control information stored in the preset local index node and the current operation status; the change information corresponding to the global metadata fed back by the metadata server is received, and the status is verified by combining it with the dimension control information stored locally; the verified change information is synchronized to the preset local index node, and the corresponding metadata fields are updated.

[0195] In one possible implementation, the generation module 305 is further configured to:

[0196] When performing data write operations or link deletion operations, operation instructions carrying dimension control information are synchronously sent to the preset backend storage; execution feedback from the preset backend storage is received; if the execution feedback indicates failure, operation instructions carrying dimension control information are resent to the preset backend storage sequentially at preset tiered time intervals, and the number of retries is recorded; when the number of retries reaches the preset retry threshold and the operation still fails, an exception log is generated and the retry operation is terminated, and the preset local index node is restored to the state before the data write operation or link deletion operation was performed based on the dimension control information.

[0197] In one possible implementation, an alignment module 308 is further included, which is specifically used for:

[0198] Record the processing result and corresponding dimension control information for each metadata log event, and generate a log processing status report; synchronize the log processing status report and the metadata of the preset local index node to the metadata server through a heartbeat interaction link; receive global metadata verification information from the metadata server, which includes the global dimension control baseline and the list of unsynchronized logs; verify the consistency between the processing result and the global status based on the verification information, and perform alignment and correction processing on inconsistent content; synchronize the unsynchronized logs and update the preset local index node.

[0199] The metadata log replay processing device provided in this embodiment can be used to execute the above-described metadata log replay processing method. Its implementation principle and technical effect are similar, and will not be described again in this embodiment.

[0200] Figure 4 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application, such as... Figure 4 As shown, the electronic device 40 includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 also includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0201] In the specific implementation process, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above method.

[0202] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0203] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0204] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0206] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0207] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0208] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0209] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0210] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0211] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0212] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0213] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0214] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0215] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for replaying metadata logs, characterized in that, include: Retrieve metadata log events; Perform local operation verification on the metadata log event; if the metadata log event is not a local operation, extract the metadata fields from the metadata log event. The metadata fields are compared with the creation time corresponding to the preset local index node; When the creation time of the metadata field is later than the creation time of the preset local index node, data processing is performed according to the operation type corresponding to the metadata log event; The file version number is updated based on a preset version increment rule, and the updated file version number is combined with the creation time to generate dimension control information. The data processing procedure is encapsulated into a transaction in a preset key-value database based on the dimension control information. Execute all instructions in the transaction of the preset key-value database and check if there are any instructions that failed to execute; If all the instructions are executed successfully, the execution result of the data processing and the dimension control information are written into the preset key-value database and the preset local index node is updated. If one or more instructions fail to execute during the process, the executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the transaction that encapsulated the data processing process into the preset key-value database.

2. The method according to claim 1, characterized in that, Data processing is performed based on the operation type corresponding to the metadata log event, including: When the operation type is a creation operation, the target path is extracted from the metadata field and the existence of the target path is checked. If the target path does not exist, directly allocate the preset local index node and initialize the metadata; If the target path exists, determine whether the target object corresponding to the target path is a file or a directory. If it is a file, delete the file link; if it is a directory, delete the empty directory. Then, allocate the preset local index node and initialize the file metadata of the preset local index node. When the operation type is a rename operation, the source path and target path are extracted from the metadata field, the subtree structure of the source path and the target path is traversed, and the write permissions of the parent directory corresponding to the target path are verified. After the verification is passed, an atomic renaming is performed, and the corresponding path association information in the preset local index node is updated; When the operation type is a delete link operation, the target path is extracted from the metadata field and the associated nodes of the target path are traversed, and the dependency relationship of the associated nodes is verified. After successful verification, delete the local file link and clean up the corresponding associated records in the preset local index node.

3. The method according to claim 1, characterized in that, Before executing all instructions in the transaction of the preset key-value database, the process also includes: The consistency between the dimension control information and the version number and creation time carried by the transaction instruction is verified. If the verification result is inconsistent, the instruction execution is determined to have failed and a rollback process is triggered. The executed operations are undone in reverse order, the occupied system resources are released, and the data is restored to the state before the transaction that encapsulated the data processing process into the preset key-value database. If the verification results are consistent, all instructions in the transaction of the preset key-value database are executed sequentially.

4. The method according to claim 1, characterized in that, After combining the updated file version number with the creation time to generate dimension control information, the method further includes: The dimension control information is written to the preset local index node for persistent storage; According to a preset heartbeat interval, a heartbeat data packet is sent to the metadata server. The heartbeat data packet contains dimension control information and current operation status stored in the preset local index node. Receive change information corresponding to global metadata fed back by the metadata server, and complete status verification by combining the dimension control information stored locally; The verified change information is synchronized to the preset local index node, and the corresponding metadata fields are updated.

5. The method according to claim 4, characterized in that, After synchronizing the verified change information to the preset local index node and updating the corresponding metadata fields, the process further includes: When performing data write operations or delete link operations, operation instructions carrying the dimension control information are simultaneously sent to the preset backend storage. Receive execution feedback from the preset backend storage; If the execution feedback indicates an execution failure, the operation instructions carrying the dimension control information are resent to the preset backend storage sequentially according to the preset stepped time intervals, and the number of retries is recorded. If the number of retries reaches the preset retry threshold and the operation still fails, an exception log is generated and the retry operation is terminated. Based on the dimension control information, the preset local index node is restored to the state before the data writing operation or the link deletion operation was performed.

6. The method according to claim 1, characterized in that, Also includes: Record the processing result and corresponding dimension control information for each metadata log event, and generate a log processing status report; The log processing status report and the metadata of the preset local index node are synchronized to the metadata server through a heartbeat interaction link; Receive verification information of global metadata from the metadata server, the verification information including global dimension control baseline and unsynchronized log list; Based on the verification information, the consistency between the processing result and the global state is verified, and the inconsistent content is aligned and corrected. Synchronize the unsynchronized logs and update the preset local index node.

7. A metadata log replay processing apparatus, characterized in that, include: The acquisition module is used to acquire metadata log events; The verification module is used to perform local operation verification on the metadata log event. If the metadata log event is not a local operation, the metadata fields in the metadata log event are extracted. The comparison module is used to compare the metadata field with the creation time corresponding to the preset local index node; The processing module is used to process data according to the operation type corresponding to the metadata log event when the creation time of the metadata field is later than the creation time of the preset local index node; The generation module is used to update the file version number based on a preset version increment rule, and combine the updated file version number with the creation time to generate dimension control information. An encapsulation module is used to encapsulate the data processing process into a transaction in a preset key-value database based on the dimension control information. The execution module is used to execute all instructions in the transaction of the preset key-value database and detect whether there are any instructions that failed to execute. The execution module is further configured to, if all the instructions are executed successfully, write the execution result of the data processing and the dimension control information into a preset key-value database and update the preset local index node; The execution module is further configured to, if one or more instructions fail to execute during the execution process, revert the executed operations in reverse order, release the occupied system resources, and restore the data to the state before the transaction that encapsulated the data processing process into the preset key-value database.

8. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images