Metadata full-amount snapshot exporting method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a metadata full-amount snapshot exporting method and device, computer equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a newly added modification serial number of metadata and an index node identification number when the metadata is modified; extracting incremental file attribute parameters from the metadata according to the newly added modification serial number and the index node identification number; combining the incremental file attribute parameter and the current memory metadata snapshot to obtain an updated memory metadata snapshot; in response to a snapshot export request sent by a user side, carrying out file path splicing according to the updated memory metadata snapshot to obtain a spliced path; and writing the splicing path into the original metadata full-amount snapshot to obtain a target metadata full-amount snapshot, and exporting the target metadata full-amount snapshot to a user side. The method can be applied to business systems needing a large amount of data, such as financial science and technology and health medical treatment, and the performance overhead and delay of metadata full-amount snapshot export can be reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applied to the fields of fintech and healthcare. In particular, it relates to a method and apparatus for exporting full metadata snapshots, computer equipment, and storage media. Background Technology

[0002] In a distributed file system, a full metadata snapshot is a complete snapshot of all metadata in memory at a specific moment. It is the core basis for the distributed file system's startup, recovery, and metadata reconciliation. In large-scale distributed scenarios, it is difficult to guarantee the global order of operation records from different nodes. Directly scanning the full metadata and exporting a full snapshot would incur significant performance overhead and latency. For example, in fintech scenarios, distributed file systems are used to achieve multiple backups to ensure data integrity, storing financial transaction data, financial statements, and transaction records. Similarly, in healthcare, patient medical imaging data, electronic medical records, and medical research data are also stored using distributed file systems to save local storage space. Therefore, for fintech and healthcare scenarios, directly scanning metadata to generate a full snapshot would impact the performance of the file distribution system, thereby affecting the efficiency of financial transactions and medical data retrieval. Therefore, reducing the performance overhead and latency of exporting full metadata snapshots has become a pressing technical problem. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, computer device, and storage medium for exporting full metadata snapshots, aiming to reduce the performance overhead and latency of exporting full metadata snapshots.

[0004] To achieve the above objectives, a first aspect of this application proposes a method for exporting a full snapshot of metadata, applied to a distributed file system connected to a user terminal, the method comprising: Obtain new and modified records of metadata and a current in-memory metadata snapshot; wherein, the new and modified records include a new and modified sequence number and an inode identifier number when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; Based on the newly added / modified serial number and the inode identifier, file attribute parameters are extracted from the metadata to obtain incremental file attribute parameters; The incremental file attribute parameters and the current memory metadata snapshot are merged to obtain an updated memory metadata snapshot; In response to a snapshot export request sent by the user, the file path is concatenated according to the updated memory metadata snapshot to obtain the concatenated path; The splicing path is written into a preset full snapshot of the original metadata to obtain a full snapshot of the target metadata, and the full snapshot of the target metadata is exported to the user terminal.

[0005] In some embodiments, the metadata includes at least one candidate file attribute parameter; the step of extracting file attribute parameters from the metadata based on the addition / modification sequence number and the inode identifier to obtain incremental file attribute parameters includes: The index node identifiers are aggregated based on the newly added or modified sequence numbers to obtain the current index node set; Obtain the running status indicators of the current index node set; The operating status indicators are compared with preset reference status indicators to obtain comparison information; The candidate file attribute parameters are filtered based on the comparison information and the inode identifier of the current inode set to obtain the incremental file attribute parameters.

[0006] In some embodiments, the running status indicators include the number of index nodes and the establishment time, and the reference status indicators include: a preset number and a preset time. The step of filtering the candidate file attribute parameters based on the comparison information and the inode identifier of the current inode set to obtain the incremental file attribute parameters includes: If the comparison information indicates that the number of index nodes has reached a preset number, or the establishment time has reached a preset time, the current index node set is converted into an updated index node set; The incremental file attribute parameters are obtained by batch acquiring the candidate file attribute parameters using a preset coroutine and the inode identifier in the updated inode set.

[0007] In some embodiments, the step of responding to a snapshot export request sent by the user client by concatenating file paths based on the updated memory metadata snapshot to obtain a concatenated path includes: In response to the snapshot export request sent by the user terminal, file path detection is performed based on the inode identifier in the updated memory metadata snapshot to obtain file path detection information; Based on the file path detection information, the parent inode identifier in the updated memory metadata snapshot is concatenated with the file path to obtain the concatenated path.

[0008] In some embodiments, the step of concatenating the parent inode identifier in the updated memory metadata snapshot with the file path detection information to obtain the concatenated path includes: If the file path detection information representing the file path of the inode identifier is empty, then the file path search is performed by tracing back upwards from the parent inode identifier in the updated memory metadata snapshot to obtain the path fragment; The path fragments are concatenated based on the parent inode identifier in the updated memory metadata snapshot to obtain the concatenated path.

[0009] In some embodiments, after merging the incremental file attribute parameters and the current memory metadata snapshot to obtain an updated memory metadata snapshot, the method further includes: The updated memory metadata snapshot is divided according to a preset time interval to obtain metadata snapshot blocks and filenames of the metadata snapshot blocks. The filenames include checkpoint information when the updated memory metadata snapshot is built. Based on the checkpoint information, the metadata snapshot is cut into chunks and exported to a preset local storage disk, so as to extract the updated memory metadata snapshot from the local storage disk in response to the snapshot export request.

[0010] In some embodiments, the distributed file system consists of at least one candidate node, and the candidate node is either a child node or a master node. The snapshot export request is responded to by the child node. Before responding to the snapshot export request sent by the user and concatenating the file path according to the updated memory metadata snapshot to obtain the concatenated path, the method further includes: Obtain the snapshot configuration parameters and current running status of the candidate nodes; Select the candidate nodes that are currently active as the selected nodes; The selected node is configured with performance evaluation based on the snapshot configuration parameters to obtain configuration performance evaluation data. Based on the configuration performance evaluation data, the selected node is divided into the master node and the child node. The master node is used to respond to the snapshot export request sent by the user.

[0011] To achieve the above objectives, a second aspect of this application provides a metadata full snapshot export device, applied to a distributed file system connected to a user terminal, the device comprising: The acquisition module is used to acquire new and modified records of metadata and the current memory metadata snapshot; wherein, the new and modified records include a new and modified sequence number and an inode identifier number when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; The parameter extraction module is used to extract file attribute parameters from the metadata based on the newly added / modified sequence number and the inode identifier number to obtain incremental file attribute parameters; The snapshot merging module is used to merge the incremental file attribute parameters and the current memory metadata snapshot to obtain an updated memory metadata snapshot; The path concatenation module is used to respond to the snapshot export request sent by the user terminal, and concatenate the file path according to the updated memory metadata snapshot to obtain the concatenated path; The snapshot export module is used to write the spliced ​​path into a preset full snapshot of the original metadata to obtain a full snapshot of the target metadata, and then export the full snapshot of the target metadata to the user terminal.

[0012] To achieve the above objectives, a third aspect of the present application provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0013] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0014] The metadata full snapshot export method, apparatus, computer device, and storage medium proposed in this application solve the problem of out-of-order modification records in distributed environments by setting a new modification sequence number constructed from the timestamp of metadata modification, and combining the new modification sequence number and the inode identifier number when the metadata was modified to form a new modification record. Then, during the metadata full snapshot export process, incremental file attribute parameters in the metadata are extracted by combining the new modification sequence number and the inode identifier number. An in-memory metadata snapshot is constructed using these incremental file attribute parameters, and upon receiving a snapshot export request, a full metadata snapshot is generated and exported using the in-memory metadata snapshot. Therefore, the full metadata snapshot is generated only using incremental file attribute parameters, eliminating the need to scan all file attribute parameters, significantly reducing the time required for full metadata snapshot export and improving its efficiency. Therefore, when applied to reconciliation and auditing in distributed file systems, only comparing the full metadata snapshots of the original and current distributed file systems is needed to accurately identify metadata differences, providing a reliable basis for system reconciliation. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method for exporting a full snapshot of metadata provided in an embodiment of this application; Figure 2 yes Figure 1 The flowchart of step S102 in the document; Figure 3 yes Figure 2 The flowchart of step S204 in the process; Figure 4 This is a flowchart of a method for exporting a full snapshot of metadata provided in another embodiment of this application; Figure 5 This is a flowchart illustrating the process of generating and exporting an updated memory metadata snapshot in the metadata full snapshot export method provided in this application embodiment; Figure 6 This is a flowchart of a method for exporting a full snapshot of metadata provided in another embodiment of this application; Figure 7 This is a flowchart of the master-child node election process in the metadata full snapshot export method provided in this application embodiment; Figure 8 yes Figure 1 The flowchart of step S104 in the process; Figure 9 yes Figure 8 The flowchart of step S802 in the process; Figure 10 This is an overall flowchart of the metadata full snapshot export method provided in the embodiments of this application; Figure 11 This is a schematic diagram of the metadata full snapshot export device provided in the embodiments of this application; Figure 12 This is a schematic diagram of the hardware structure of the computer device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0017] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0019] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0020] Hadoop Distributed File System (HDFS) is a file system for managing large datasets that runs on commercial hardware. HDFS is the most commonly used data storage system for Hadoop, capable of scaling a single Apache Hadoop cluster to hundreds or even thousands of nodes. Because HDFS can efficiently manage large amounts of data with high throughput, it can be used as a data pipeline and is well-suited for supporting complex data analysis.

[0021] Metadata, also known as intermediary data or relay data, is data about data. It primarily describes data properties and supports functions such as indicating storage location, historical data, resource lookup, and file records. Metadata is a type of electronic catalog; to achieve the purpose of cataloging, it is necessary to describe and collect the content or characteristics of the data, thereby assisting in data retrieval.

[0022] Incremental snapshots are a key technology in the field of data storage and backup. The core of this technology is to record only the data blocks that change between two snapshots, rather than repeatedly storing the entire dataset. Compared with "full snapshots" (which back up the entire data each time), this can significantly save storage space, shorten backup time, and reduce IO overhead.

[0023] Full metadata snapshot (FSImage): It is a core component of the Hadoop Distributed File System (HDFS). Essentially, it is a complete and persistent snapshot of HDFS metadata (file / directory structure, permissions, block mapping relationships, etc.) at a certain point in time. FSImage is the "full baseline snapshot" of HDFS, which, together with the incremental log (EditLog), enables efficient persistence and recovery of metadata.

[0024] Global Transaction ID (GTID): A globally unique, monotonically increasing identifier used to uniquely identify each committed transaction in a database cluster.

[0025] TiKV (TiKV), an open-source distributed key-value database, is designed to build scalable distributed databases and address performance, reliability, and scalability issues in large-scale distributed storage scenarios. TiKV is based on the Raft consensus algorithm, ensuring data consistency and high availability, and employs horizontal scaling, allowing the system to scale linearly with the amount of data.

[0026] Coroutines are user-mode threads, but the timing of context switching is controlled by the caller. Coroutines are very similar to user-mode threads; switching between user-mode threads does not require entering the kernel, but in some operating systems, switching between user-mode threads requires the assistance of kernel-mode threads.

[0027] An inode is a structure in a file system used to store file metadata. Each file has a corresponding inode, which contains basic information about the file, but not its name or actual data content. The inode is the file's index node, responsible for storing all information related to the file, while the filename is a reference to an inode.

[0028] In a distributed file system, the method and system mainly utilize TiKV transaction global sequence numbers to achieve operation order preservation, build memory images, and generate full snapshots of metadata. It is applicable to distributed file systems compatible with the HDFS protocol, such as Hadoop Distributed File System and OBS-HDFS.

[0029] In a distributed file system, a full metadata snapshot is a complete snapshot of all metadata in memory at a given moment, serving as the core basis for distributed file system startup, recovery, and metadata reconciliation. However, in large-scale distributed scenarios with a multi-NameNode architecture employing gateway-layer load balancing, it is difficult to guarantee the global order of modification records from different nodes. Directly scanning the entire metadata to generate a full metadata snapshot would incur significant performance overhead and latency. For example, in fintech, distributed file systems can achieve distributed storage of transaction and user data in the insurance and banking sectors. In healthcare, distributed file systems can achieve distributed storage of medical image files, patient records, and other data, saving local storage space. Therefore, the latency caused by exporting full metadata snapshots impacts data storage and retrieval across various domains, necessitating a metadata snapshot generation mechanism that can guarantee the order of modifications in a distributed environment, support efficient incremental generation of full metadata snapshots, and possess rapid recovery capabilities.

[0030] Based on this, this application provides a method and apparatus, computer device, and storage medium for exporting a full metadata snapshot. The aim is to obtain new and modified records of metadata and a current memory metadata snapshot. The new and modified records include a new / modified sequence number and an inode identifier, with the sequence number constructed from the timestamp of the metadata modification, thus enabling sequential setting of metadata modification records. Therefore, incremental file attribute parameters of the metadata are extracted based on the new / modified sequence number and inode identifier. These parameters are then merged with the current memory metadata snapshot to form an updated memory metadata snapshot. Upon receiving a snapshot export request from the user, the file paths corresponding to the updated memory metadata snapshot are concatenated into a concatenated path, which is then written into the original full metadata snapshot to obtain the target full metadata snapshot. Finally, the re-exported target metadata snapshot is sent to the user. Therefore, this embodiment only processes incremental file attribute parameters, completing the construction of the full metadata snapshot through these parameters. This eliminates the need to rescan the full metadata for each full snapshot export, reducing metadata pressure and generation time, and improving the response efficiency of the distributed file system.

[0031] The metadata full snapshot export method, apparatus, computer device, and storage medium provided in this application are specifically described through the following embodiments. First, the metadata full snapshot export method in this application embodiment is described.

[0032] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] The metadata full snapshot export method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the metadata full snapshot export method, but is not limited to the above forms.

[0035] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0036] Figure 1 This is an optional flowchart of the metadata full snapshot export method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S105.

[0037] Step S101: Obtain the new and modified records of metadata and the current memory metadata snapshot; wherein, the new and modified records include the new and modified sequence number and the inode identifier number when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; Step S102: Extract file attribute parameters from metadata based on the newly added / modified sequence number and inode identifier to obtain incremental file attribute parameters; Step S103: Merge the incremental file attribute parameters and the current memory metadata snapshot to obtain the updated memory metadata snapshot; Step S104: In response to the snapshot export request sent by the user, the file path is concatenated according to the updated memory metadata snapshot to obtain the concatenated path; Step S105: Write the splicing path into the preset full snapshot of the original metadata to obtain the full snapshot of the target metadata, and export the full snapshot of the target metadata to the user terminal.

[0038] Steps S101 to S105 of this embodiment involve constructing a new modification sequence number from the timestamp when the metadata is modified, and combining the new modification sequence number with the inode identifier of the modified metadata to form a new modification record. This ensures that metadata modification operations are performed in an orderly manner, facilitating the orderly export of the full metadata snapshot. Simultaneously, during the full metadata snapshot export, incremental file attribute parameters are extracted from the metadata based on the new modification sequence number and inode identifier, and these parameters are merged into the current in-memory metadata snapshot to obtain an updated in-memory metadata snapshot. Then, upon receiving a snapshot export request, file paths are concatenated into a concatenated path based on the updated in-memory metadata snapshot, and this concatenated path is written into the original full metadata snapshot to form the target full metadata snapshot. Finally, the target full metadata snapshot is exported to the user. Therefore, combining in-memory mirroring and incremental merging mechanisms to generate a full metadata snapshot during the full metadata snapshot export process can improve the availability and reconciliation accuracy of the distributed file system, reduce performance overhead and latency, and increase the efficiency of full metadata snapshot generation.

[0039] In step S101 of some embodiments, metadata is data that describes data, mainly describing information about data attributes to support functions such as indicating storage location, historical data, resource lookup, and file records. Metadata includes candidate file attribute parameters, also known as Attrs, which are the core components of metadata. Candidate file attribute parameters are the smallest descriptive unit of metadata, providing a clear definition of a specific characteristic of a file / directory (such as "modification time," "file size," "permissions," etc.).

[0040] In this embodiment, all metadata modification operations are written to the open-source database as new modification records. Each new modification record includes a new modification sequence number, constructed using the timestamp of when the metadata was modified. This sequence number serves as the global sequence number for the modification operation, generating a globally ordered set of new modification records. It should be noted that metadata modification operations include creation, deletion, and renaming operations, etc., and this embodiment does not limit the types of metadata modification operations. Furthermore, the metadata modification operations are stored in the open-source database according to the transaction initiation timestamp based on the new modification sequence number, forming a global new modification record. Therefore, by scanning a specified set of new modification records, the change sequence of the entire directory tree can be obtained.

[0041] Specifically, a new modification record includes a new / modification sequence number, a namespace identifier, an inode identifier when the metadata was modified, and a parent inode identifier. The new / modification sequence number is defined as TSO, the namespace identifier as NamespaceID, the inode identifier as Inode, and the parent inode identifier as ParentInode. Therefore, the format of a new / modification record is as follows: <prefix> _ <tso><NamespaceID, Inode,ParentInode>For example, file F1 was added in TSO1, and directory DIR1 was added in TSO2. By recording the modification operations of metadata according to the addition and modification sequence number, it is possible to achieve globally ordered modification operation recording, so as to accurately reorganize the directory tree at any point in time in the entire distributed file system based on the addition and modification sequence number in the addition and modification record.

[0042] Please see Figure 2 In some embodiments, step S102 may include, but is not limited to, steps S201 to S204: Step S201: Summarize the index node identifiers according to the newly added or modified sequence numbers to obtain the current index node set; Step S202: Obtain the running status indicators of the current index node set; Step S203: Compare the running status indicators with the preset reference status indicators to obtain comparison information; Step S204: Based on the comparison information and the inode identifier of the current inode set, the candidate file attribute parameters are filtered to obtain the incremental file attribute parameters.

[0043] In step S201 of some embodiments, the inode identifier, also known as an inode, is a unique number assigned to each file or directory in the distributed file system. The inode identifier clearly identifies which file or directory has been modified. Therefore, the inode identifiers are aggregated into a current inode set according to the sequence number of additions and modifications. Specifically, addition and modification records are scanned from the open-source database, and the inode identifiers that changed during the addition and modification operations are obtained and aggregated to obtain the current inode set. It should be noted that the current inode set represents all the inode identifiers that have changed and is defined as CurrentSet. It should also be noted that the current inode set includes the operation log of the distributed file system within a preset time period. Therefore, the current inode set allows for an orderly view of which metadata has undergone modification operations, serving as a core element for the startup, recovery, and metadata reconciliation of the distributed file system.

[0044] Specifically, the first coroutine, defined as coroutine A, is started. Coroutine A uses checkpoints as the starting boundary to scan for newly added and modified records in the open-source database, and obtains the sequence numbers of all the newly added and modified records that have changed to form the current set of index nodes.

[0045] In steps S202 and S203 of some embodiments, the running status indicator represents the creation time and number of indexes of the current inode set, and the reference status indicator is used to evaluate whether the creation time of the current inode set meets the set time limit, or whether the number of indexes meets the preset quantity requirement. Therefore, comparison information is obtained by comparing the running status indicator and the reference status indicator, and the comparison information is used to determine whether the current inode set can update the current metadata memory snapshot.

[0046] In step S204 of some embodiments, before updating the current metadata memory snapshot, it is necessary to extract the candidate file attribute parameters corresponding to the changed inode identifiers as incremental file attribute parameters to determine which file attribute parameters have changed. It should be noted that after the metadata modification operation is recorded, the incremental file attribute parameters are combined according to the changed candidate file attribute parameters. It is not necessary to scan all candidate file attribute parameters of the metadata before constructing the memory metadata snapshot. The update of the current memory metadata snapshot is completed through incremental file attribute parameters, improving the efficiency of exporting the full metadata snapshot.

[0047] In steps S201 to S204 of this embodiment, the inode identifiers of the newly added and modified sequence numbers are combined into the current inode set, and the running status indicators of the current inode set are compared with the reference status indicators to determine comparison information. Finally, based on the comparison information and the inode identifiers of the current inode set, incremental file attribute parameters are selected from the candidate file attribute parameters to accurately select incremental file attribute parameters and quickly and accurately complete the update of the current metadata memory snapshot.

[0048] In some embodiments, the running status indicators include: the number of index nodes and the establishment time, and the reference status indicators include: the preset number and the preset time.

[0049] Please see Figure 3 In some embodiments, step S204 may include, but is not limited to, steps S301 to S302: Step S301: If the comparison information indicates that the number of index nodes has reached a preset number, or the establishment time has reached a preset time, the current index node set is converted into an updated index node set. Step S302: The candidate file attribute parameters are obtained in batches by using a preset coroutine and the inode identifier in the updated inode set to obtain incremental file attribute parameters.

[0050] In step S301 of some embodiments, if the number of index nodes in the current index node set reaches a preset threshold, or the establishment time of the current index node set reaches a preset time, the indicator representing the running status of the current index node set meets the preset conditions. Defining the current index node set as an update index node set is equivalent to converting CurrentSet to FlushSet. It should be noted that defining the current index node set that meets the preset conditions as an update index node set, and continuing to summarize the changed index node identifiers to form a new current index node set, is to distinguish between the update index node set that meets the preset conditions and the current index node set that does not meet the preset conditions.

[0051] In step S302 of some embodiments, batch retrieval involves retrieving candidate file attribute parameters one by one from the updated index node set according to the order of the new / modified sequence number, as incremental file attribute parameters, thereby reducing the number of accesses to the open-source database. Furthermore, the retrieval of incremental file attribute parameters is also accomplished through a second coroutine, defined as coroutine B. Coroutine B performs the batch retrieval operation, retrieving candidate file attribute parameters corresponding to each index node identifier in the updated index node set from the open-source database as incremental file attribute parameters, i.e., incremental file / directory information.

[0052] In steps S301 to S302 of this embodiment, by detecting that the number of index nodes in the current index node set has reached a preset number, or the establishment time has reached a preset time, the current index node set is directly converted into an updated index node set. Incremental file attribute parameters are obtained by batch retrieving the candidate file attribute parameters corresponding to the index node identifiers of the updated index node set through a coroutine. On the one hand, this reduces the number of accesses to the open source database, and on the other hand, it only retrieves the incremental file attribute parameters corresponding to the changed index node identifiers, which can improve the update efficiency of the current metadata memory snapshot, thereby improving the generation efficiency of the full metadata snapshot.

[0053] In step S103 of some embodiments, the current memory metadata snapshot is a snapshot of the memory metadata in memory at the current time. It is a data structure that, at a specific point in time, completely records the directory tree structure of the distributed file system in memory and the metadata of each element (file / directory). Therefore, the directory tree structure of the distributed file system can be determined through the current memory metadata snapshot. Specifically, merging the incremental file attribute parameters into the current memory metadata snapshot mainly involves three steps: First, creating an updated memory metadata snapshot and copying the complete directory tree structure and metadata of the current memory metadata snapshot into the updated memory metadata snapshot, while also copying the inode identifier of the current memory metadata snapshot; Second, iterating over each incremental file attribute parameter; Third, searching for the corresponding node in the updated memory metadata snapshot using the inode identifier of the incremental file attribute parameter. If the incremental file attribute parameter records a deletion operation, the corresponding inode identifier is removed from the updated memory metadata snapshot; if it is a modification, the metadata corresponding to the inode identifier is updated using the incremental file attribute parameter. By gradually completing the inode location and content update of each incremental file attribute parameter, the incremental file attribute parameters and the current memory metadata snapshot are merged to form an updated memory metadata snapshot.

[0054] Please see Figure 4 In some embodiments, after step S103, the metadata full snapshot export method may also include, but is not limited to, steps S401 to S402: Step S401: The updated memory metadata snapshot is divided according to a preset time interval to obtain metadata snapshot chunks and filenames of metadata snapshot chunks. The filenames include checkpoint information when the updated memory metadata snapshot is built. Step S402: Based on the checkpoint information, export the metadata snapshot chunks to the preset local storage disk, so as to extract the updated memory metadata snapshot from the local storage disk in response to the snapshot export request.

[0055] In step S401 of some embodiments, the splitting involves dividing the updated memory metadata snapshot into multiple slices / blocks, also known as splitting the current directory tree information into multiple slices / blocks. Each slice / block is defined as a metadata snapshot block. The setting of metadata snapshot blocks allows for fast export, saving export time. The filename of the metadata snapshot block contains checkpoint information, which represents the sequence number of the currently scanned new and modified records. This helps to determine the point in time when the new and modified records were added or modified. After restarting the distributed file system, the next new and modified record can be scanned based on the checkpoint information in the filename of the metadata snapshot block, thus completing the continuous updating of the current memory metadata snapshot.

[0056] In step S402 of some embodiments, the metadata snapshot is divided into blocks and exported to the local storage disk according to the checkpoint information. The updated memory metadata snapshot is exported in a block-based manner, which can improve the export efficiency.

[0057] Specifically, the checkpoint information is defined as checkpoint, for example, memsnap_{checkpoint}.part. The metadata snapshot is divided into blocks according to the checkpoint information and exported to the local disk in parallel, or the metadata snapshot is divided into blocks and exported to the cloud storage disk, such as OBS-HDFS.

[0058] In this embodiment, the updated memory metadata snapshots are stored in blocks on the local storage disk to form memory image files. If the distributed file system restarts, the most recent memory image file is read from the local storage disk to restore the updated memory metadata snapshots. After restoration, snapshot updates are continued for new and modified records after the checkpoint information, in order to build an updated memory metadata database that perfectly matches the actual data situation of the distributed file system.

[0059] In steps S401 to S402 of this embodiment, the updated memory metadata snapshot is periodically exported to the current storage disk in a block format, which improves the export efficiency of the updated memory metadata snapshot and realizes the persistence and recovery of the memory image.

[0060] Please refer to Figure 5 , Figure 5 This illustrates the process of generating and exporting an updated in-memory metadata snapshot in the full metadata snapshot export method. In the distributed file system, all metadata modifications are written to an open-source database, specifically the TIKV database, as new modification records. It's important to note that the timestamp of the metadata modification is used as the new modification sequence number, and the inode identifier at the time of the metadata modification is stored in the open-source database according to this sequence number. Then, through coroutine A, using checkpoint information as the starting boundary, the inode identifiers in the new modification records are scanned from the TIKV database to obtain the changed current inode set, defined as CurrentSet. If the number of inodes in the current inode set does not reach a preset number, or the creation time does not reach a preset duration, the current inode set continues to be updated. If the number of inodes in the current inode set reaches the preset number, or the creation time reaches the preset duration, the current inode set is converted into an updated inode set, defined as FlushSet. Then, the candidate file attribute parameters corresponding to all inode identifiers in the updated inode set are obtained as incremental file attribute parameters, defined as IncInodeMap. These incremental file attribute parameters are then merged with the current memory metadata snapshot, which is defined as MemSnapMap, to obtain the updated memory metadata snapshot, i.e., the full directory tree information at that moment. To achieve memory image persistence and recovery, the updated memory metadata snapshot is stored in blocks on the local storage disk or OBS-HDFS, and previous versions of the current memory metadata snapshot are deleted. Therefore, by using the addition / modification sequence number constructed with the modification timestamp and the inode identifier as a globally ordered addition / modification record, the problem of out-of-order modification operation records in the distributed file system is solved. Simultaneously, by avoiding a full scan and only processing incremental file attribute parameters, and merging the incremental file attribute parameters with the current memory metadata snapshot to form the updated memory metadata snapshot, metadata pressure and latency in the memory metadata snapshot generation process are reduced.

[0061] In some embodiments, the distributed file system includes at least one candidate node, and the candidate node is divided into a master node and at least one child node. It should be noted that the master node is used to export a full snapshot of the metadata, while the child nodes are only used to transmit snapshot export requests.

[0062] Please see Figure 6 In some embodiments, prior to step S104, the metadata full snapshot export method may also include, but is not limited to, steps S601 to S604: Step S601: Obtain the snapshot configuration parameters and current running status of the candidate nodes; Step S602: Select the candidate nodes that are currently active as the selected nodes; Step S603: Perform configuration performance evaluation on the selected node based on the snapshot configuration parameters to obtain configuration performance evaluation data; Step S604: Based on the configuration performance evaluation data, the selected node is divided into a master node and a child node. The master node is used to respond to the snapshot export request sent by the user.

[0063] In step S601 of some embodiments, snapshot configuration parameters are used to evaluate the storage performance of each candidate node, and the snapshot configuration parameters are ETCD leases and key-value pairs, using a key of the candidate node, a value of the candidate node's address, and setting the lease. The current running state includes an active state and a dormant state, where the active state indicates that the candidate node is still running, and the dormant state indicates that the candidate node is not running.

[0064] In steps S602 and S603 of some embodiments, candidate nodes that are currently in an active state are selected as selected nodes, and the selected nodes are evaluated for performance based on the snapshot configuration parameters of the selected nodes to obtain configuration performance evaluation data, so as to determine whether the selected nodes can be used as master nodes through the configuration performance evaluation data.

[0065] In step S604 of some embodiments, as disclosed above, if the configuration performance evaluation data is represented as the key of the candidate node, defined as / hdfs / fsimage / master, if the key of the selected node exists, the selected node will be the master node; if the key does not exist, the selected node will be the child node. Simultaneously, the key of each node will be monitored in real time; if a key is deleted, a new master node will be elected. It should be noted that the selected master node is defined to respond to snapshot export requests, while child nodes only forward snapshot export requests. Specifically, if the master node receives a snapshot export request, it directly responds to the request and completes a full snapshot export of the metadata. If a child node receives a snapshot export request, it forwards the request to the master node, which then completes the full snapshot export of the metadata.

[0066] In this embodiment, the master node and the child node share the underlying open-source database and OBS-HDFS storage, eliminating the need for data transfer between the master and child nodes. This ensures data consistency and reduces the risk of data security issues caused by data transfer.

[0067] like Figure 7 As shown, Figure 7 The diagram illustrates the master and child node election mechanism. Each candidate node is essentially a server in the distributed file system. If server 1 and server 2 start, they respectively restore memory metadata snapshots from an open-source database or a memory image file. Then, server 1 and server 2 begin master node election. If server 1 has the key `hdfs / fsimage / master`, it is defined as the master node; if server 2 does not have the key, it is defined as a child node. Both the master and child nodes can perform the aforementioned generation and export of updated memory metadata.

[0068] In step S104 of some embodiments, the snapshot export request is responded to by the master node, which mainly completes path concatenation and snapshot output. For path concatenation, it refers to completing path concatenation according to the identifier number of each inode and the identifier number of the parent inode in the updated memory metadata snapshot to construct a complete path.

[0069] Please see Figure 8 In some embodiments, step S104 includes, but is not limited to, steps S801 to S802: Step S801: In response to the snapshot export request sent by the user, file path detection is performed based on the inode identifier in the updated memory metadata snapshot to obtain file path detection information; Step S802: Based on the file path detection information, the parent inode identifier in the updated memory metadata snapshot is concatenated with the file path to obtain the concatenated path.

[0070] In step S801 of some embodiments, path concatenation mainly involves constructing a complete path by backtracking the parent node through each inode identifier in the updated memory metadata snapshot, and file path detection mainly involves checking whether the path corresponding to the inode identifier in the updated memory metadata snapshot is empty. Therefore, the file path detection information characterizes whether the access path of the file / directory for each inode identifier is empty.

[0071] In step S802 of some embodiments, the parent inode identifier in the updated memory metadata snapshot is concatenated with the file path based on the file path detection information. Specifically, the path is found first and then concatenated to form a complete path.

[0072] In steps S801 to S802 of this embodiment, when responding to a snapshot export request sent by the user, the system first checks whether a corresponding path exists based on the index node identifier in the updated memory metadata snapshot, and then combines the parent index node of the updated memory metadata snapshot to complete the path splicing, so as to construct a complete spliced ​​path.

[0073] Please see Figure 9 In some embodiments, step S802 may include, but is not limited to, steps S901 to S902: Step S901: If the file path representing the inode identifier in the file path detection information is empty, then the file path is searched by backtracking upwards from the parent inode identifier in the updated memory metadata snapshot to obtain the path fragment. Step S902: Concatenate the path fragments according to the parent inode identifier in the updated memory metadata snapshot to obtain the concatenated path.

[0074] In step S901 of some embodiments, if the file path detection information indicates that the file path corresponding to the inode identifier is empty, then the process starts from the parent inode identifier in the updated memory metadata snapshot and traces back upwards, sequentially obtaining the name of the parent inode identifier and the parent node, until the root inode is reached to determine the path segment. Gradually, the path detection and upward tracing continue according to the inode identifier and parent inode identifier in the updated memory metadata snapshot to obtain at least one path segment.

[0075] In step S902 of some embodiments, path fragments are concatenated in reverse order to obtain a concatenated path. For example, if the path / A / B / C.TXT is found directly based on the inode identifier, then the path / A / B / C.TXT is directly output as the concatenated path. If the path C.TXT is found based on the inode identifier, then this path is defined as empty at this time, so B must be found first, then A, and finally the root directory, concatenating a path of / A / B / C.TXT. It should be noted that during the path concatenation process, since the file attribute parameters in the updated memory source database only contain the names of the current file and directory, but not the path, the path needs to be traced back upwards to find it.

[0076] In steps S901 to S902 of this embodiment, a complete splicing path is constructed by backtracking the parent node for each index node identifier of the updated memory metadata snapshot, so that the candidate can accurately generate a full metadata snapshot.

[0077] In step S105 of some embodiments, the original full metadata snapshot is also called FSImage, which represents a snapshot image of the previous full metadata, that is, the directory tree structure and file / directory metadata before the metadata was modified. Therefore, all the concatenated paths are directly written into the original full metadata snapshot to form an updated target metadata full snapshot, and exported to the user end to achieve fast export of the target metadata full snapshot.

[0078] Please refer to Figure 10 This application's embodiments can be applied to distributed file systems in the fintech and healthcare fields. Taking insurance business scenarios as an example, a large amount of policy data needs to be stored in a distributed file system, and this policy data is updated and accessed in real time, so the metadata within the distributed file system is also updated synchronously. However, after the distributed file system restarts, it needs to be restored in real time, which requires exporting a full snapshot of the previous metadata to complete the restoration of the distributed file system. This embodiment sets up a method for exporting a full snapshot of metadata for a distributed file system, which can export quickly and with low latency. The overall process is described in detail below: Step S1001, Storage and scanning of new and modified records: Use the open-source database TiKV to store new and modified records, and the format of the new and modified records is as follows: <prefix> _ <tso><NamespaceID,Inode, ParentInode>Among them, TSO is the addition / modification sequence number set according to the timestamp when the metadata was modified, NamespaceID is the namespace ID, Inode is the inode identifier number when the metadata was modified, and ParentInode is the parent inode identifier number.

[0079] Step S1002, memory image construction: Coroutine A scans for new and modified records, and coroutine B batch obtains the candidate file attribute parameters corresponding to the inode identifier in the new and modified records and updates the current memory metadata snapshot to obtain the updated memory metadata snapshot.

[0080] Step S1003, Memory Image Persistence and Recovery: Periodically export updated memory metadata snapshots in block form to local storage disk or OBS-HDFS, and support distributed file system recovery from the exported block files.

[0081] Step S1004, FSImage generation: The master node updates the identifier of each index node in the memory metadata snapshot, constructs the complete splicing path by tracing back the parent node, and then exports it as a CSV file.

[0082] In summary, by using timestamps as the sequence number for new modifications, a globally ordered record of new and modified data is constructed, completely resolving the issue of out-of-order modification records in a distributed environment. Simultaneously, incremental file attribute parameters are set to update the current in-memory metadata snapshot, avoiding a full scan of all candidate file attribute parameters and processing only incremental file attribute parameters, significantly reducing metadata pressure and the time required to generate in-memory metadata snapshots. Block-based export and recovery of updated in-memory metadata snapshots are implemented, allowing for rapid state recovery after service restarts in the distributed file system, significantly shortening recovery time. The snapshot export process is completed by the master node, and the filtering mechanism of the master and child nodes, combined with ETCD, ensures the continuous availability of the full metadata snapshot generation service. Therefore, by exporting the target metadata full snapshot, only a comparison between the target metadata full snapshot exported from the source HDFS and the target HDFS is needed to accurately identify metadata differences, providing a reliable basis for distributed file system reconciliation.

[0083] Please see Figure 11 This application also provides a metadata full snapshot export device, which can implement the above-described metadata full snapshot export method and is applied to a distributed file system connected to a user terminal. The device includes: The acquisition module 1101 is used to acquire new and modified records of metadata and the current memory metadata snapshot; wherein, the new and modified records include the new and modified sequence number and the inode identifier number when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; The parameter extraction module 1102 is used to extract file attribute parameters from metadata based on the newly added / modified sequence number and inode identifier number to obtain incremental file attribute parameters. The snapshot merging module 1103 is used to merge the incremental file attribute parameters and the current memory metadata snapshot to obtain an updated memory metadata snapshot; The path concatenation module 1104 is used to respond to the snapshot export request sent by the user terminal, and concatenate the file path according to the updated memory metadata snapshot to obtain the concatenated path; The snapshot export module 1105 is used to write the spliced ​​path into a preset full snapshot of the original metadata to obtain a full snapshot of the target metadata, and then export the full snapshot of the target metadata to the user terminal.

[0084] The specific implementation of the metadata full snapshot export device is basically the same as the specific implementation of the metadata full snapshot export method described above, and will not be repeated here.

[0085] This application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for exporting a full snapshot of metadata. This computer device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0086] Please see Figure 12 , Figure 12 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 1201 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 1202 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1202 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202, and the processor 1201 calls and executes the metadata full snapshot export method of the embodiments of this application. The input / output interface 1203 is used to implement information input and output; The communication interface 1204 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1205 transmits information between various components of the device (e.g., processor 1201, memory 1202, input / output interface 1203, and communication interface 1204); The processor 1201, memory 1202, input / output interface 1203 and communication interface 1204 are connected to each other within the device via bus 1205.

[0087] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for exporting full metadata snapshots.

[0088] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] The metadata full snapshot export method, apparatus, computer device, and storage medium provided in this application solve the problem of out-of-order modification records in a distributed environment by setting a new modification sequence number constructed from the timestamp of metadata modification and combining the new modification sequence number and the inode identifier when the metadata was modified to form a new modification record. Then, during the metadata full snapshot export process, incremental file attribute parameters in the metadata are extracted by combining the new modification sequence number and the inode identifier, and a memory metadata snapshot is constructed using these incremental file attribute parameters. Upon receiving a snapshot export request, the full metadata snapshot is generated and exported using the memory metadata snapshot. Therefore, the full metadata snapshot is generated only using incremental file attribute parameters, eliminating the need to scan all file attribute parameters, significantly reducing the time required for full metadata snapshot export and improving its efficiency. Therefore, when applied to reconciliation and auditing in a distributed file system, only comparing the full metadata snapshots of the original and current distributed file systems is needed to accurately identify metadata differences and provide a reliable basis for system reconciliation.

[0090] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0091] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0094] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0095] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0097] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0099] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0100] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.< / tso> < / prefix> < / tso> < / prefix>

Claims

1. A method for exporting full metadata snapshots, characterized in that, Applied to a distributed file system connected to a user terminal, the method includes: Obtain new and modified records of metadata and a current in-memory metadata snapshot; wherein, the new and modified records include a new and modified sequence number and an inode identifier when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; Based on the newly added / modified serial number and the inode identifier, file attribute parameters are extracted from the metadata to obtain incremental file attribute parameters; The incremental file attribute parameters and the current memory metadata snapshot are merged to obtain an updated memory metadata snapshot; In response to a snapshot export request sent by the user, the file path is concatenated according to the updated memory metadata snapshot to obtain the concatenated path; The splicing path is written into a preset full snapshot of the original metadata to obtain a full snapshot of the target metadata, and the full snapshot of the target metadata is exported to the user terminal.

2. The method according to claim 1, characterized in that, The metadata includes at least one candidate file attribute parameter; the step of extracting file attribute parameters from the metadata based on the new / modified sequence number and the inode identifier to obtain incremental file attribute parameters includes: The index node identifiers are aggregated based on the newly added or modified sequence numbers to obtain the current index node set; Obtain the running status indicators of the current index node set; The operating status indicators are compared with preset reference status indicators to obtain comparison information; The candidate file attribute parameters are filtered based on the comparison information and the inode identifier of the current inode set to obtain the incremental file attribute parameters.

3. The method according to claim 2, characterized in that, The operational status indicators include the number of index nodes and the establishment time, and the reference status indicators include: preset quantity and preset time. The step of filtering the candidate file attribute parameters based on the comparison information and the inode identifier of the current inode set to obtain the incremental file attribute parameters includes: If the comparison information indicates that the number of index nodes has reached a preset number, or the establishment time has reached a preset time, the current index node set is converted into an updated index node set; The incremental file attribute parameters are obtained by batch acquiring the candidate file attribute parameters using a preset coroutine and the inode identifier in the updated inode set.

4. The method according to claim 1, characterized in that, The response to the snapshot export request sent by the user client involves concatenating file paths based on the updated memory metadata snapshot to obtain the concatenated path, including: In response to the snapshot export request sent by the user terminal, file path detection is performed based on the inode identifier in the updated memory metadata snapshot to obtain file path detection information; Based on the file path detection information, the parent inode identifier in the updated memory metadata snapshot is concatenated with the file path to obtain the concatenated path.

5. The method according to claim 4, characterized in that, The step of concatenating the parent inode identifier in the updated memory metadata snapshot according to the file path detection information to obtain the concatenated path includes: If the file path detection information representing the file path of the inode identifier is empty, then the file path search is performed by tracing back upwards from the parent inode identifier in the updated memory metadata snapshot to obtain the path fragment; The path fragments are concatenated based on the parent inode identifier in the updated memory metadata snapshot to obtain the concatenated path.

6. The method according to any one of claims 1 to 5, characterized in that, After merging the incremental file attribute parameters and the current memory metadata snapshot to obtain an updated memory metadata snapshot, the method further includes: The updated memory metadata snapshot is divided according to a preset time interval to obtain metadata snapshot blocks and filenames of the metadata snapshot blocks. The filenames include checkpoint information when the updated memory metadata snapshot is built. Based on the checkpoint information, the metadata snapshot is sliced ​​and exported to a preset local storage disk, so as to extract the updated memory metadata snapshot from the local storage disk in response to the snapshot export request.

7. The method according to claim 6, characterized in that, The distributed file system consists of at least one candidate node, which can be a child node or a master node. The snapshot export request is responded to by the child node. Before responding to the snapshot export request sent by the user and concatenating the file path according to the updated memory metadata snapshot to obtain the concatenated path, the method further includes: Obtain the snapshot configuration parameters and current running status of the candidate nodes; Select the candidate nodes that are currently active as the selected nodes; The selected node is configured with performance evaluation based on the snapshot configuration parameters to obtain configuration performance evaluation data. Based on the configuration performance evaluation data, the selected node is divided into the master node and the child node. The master node is used to respond to the snapshot export request sent by the user.

8. A metadata full snapshot export device, characterized in that, The device is applied to a distributed file system that connects to a user terminal, and includes: The acquisition module is used to acquire new and modified records of metadata and the current memory metadata snapshot; wherein, the new and modified records include a new and modified sequence number and an index node identifier when the metadata was modified, and the new and modified sequence number is constructed based on the timestamp when the metadata was modified; The parameter extraction module is used to extract file attribute parameters from the metadata based on the newly added / modified sequence number and the inode identifier number to obtain incremental file attribute parameters; The snapshot merging module is used to merge the incremental file attribute parameters and the current memory metadata snapshot to obtain an updated memory metadata snapshot; The path concatenation module is used to respond to the snapshot export request sent by the user terminal, and concatenate the file path according to the updated memory metadata snapshot to obtain the concatenated path; The snapshot export module is used to write the spliced ​​path into a preset full snapshot of the original metadata to obtain a full snapshot of the target metadata, and then export the full snapshot of the target metadata to the user terminal.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the metadata full snapshot export method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the metadata full snapshot export method according to any one of claims 1 to 7.

Images