Metadata real-time monitoring method, device and equipment for distributed storage system

Abstract

The invention relates to the technical field of distributed storage system monitoring, and particularly discloses a metadata real-time monitoring method, device and equipment of a distributed storage system. The method comprises the following steps of: dynamically loading a file system mirror image to construct an initial memory metadata model by deploying an auditing service which is physically isolated from a main node and a standby node; subscribing metadata change event streams published by log nodes in real time, and incrementally updating the model after time sequence sorting and conflict detection; key dimension fields are extracted and converted into a structured format, and after batch compression, the key dimension fields are synchronized to an index engine to construct a dual-path index; and generating cross-level directory topology analysis, a user behavior portrait and storage block distribution early warning in response to query. According to the method, the problems of high time delay, metadata splitting and main node performance interference caused by offline analysis in a traditional scheme are solved, second-level updated full-quantity metadata acquisition and multi-dimensional real-time analysis are realized, and the monitoring efficiency and storage optimization capability of a massive file system are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage system monitoring, in particular to a metadata real-time monitoring method, device and equipment of a distributed storage system. BACKGROUND

[0002] In a distributed file system (such as HDFS), metadata is stored in the NameNode memory, including file path, permission, data block distribution and other key information. With the growth of big data application scale, massive small files and deep directory structure cause metadata to expand rapidly, and the single NameNode instance memory consumption can reach TB level, becoming a system performance bottleneck. The specific performance is as follows: a large number of small files cause the NameNode memory occupancy to grow nonlinearly, triggering frequent Full GC and causing service interruption; the traditional solution relies on offline analysis of FsImage files to obtain metadata snapshots, but the analysis time consumption and data volume grow exponentially, lacking timeliness; the process of converting FsImage to text loses key fields such as extended attributes and storage strategies, causing storage optimization strategies to fail; existing tools cannot construct directory tree topology and user behavior portraits in real time, making it difficult to actively identify deep redundant directories, storage quota overruns and replica distribution imbalance problems.

[0003] The current solution has fundamental defects: offline analysis mode causes problem discovery delay and cannot capture transient metadata anomalies; master node coupling design causes I / O competition when performing large-scale scanning, causing business delay; metadata storage is fragmented, EditLog logs are not used for real-time analysis, FsImage snapshots are separated from dynamic change data, causing inconsistent states.

[0004] Therefore, a metadata real-time monitoring solution that is decoupled from production services and supports second-level updates is needed to achieve multi-dimensional analysis of directory topology, user behavior and storage optimization without performance loss. SUMMARY

[0005] The technical problem to be solved by the present application is how to achieve second-level collection and multi-dimensional real-time analysis of massive metadata without interfering with the performance of the master node of the distributed file system, to overcome the high delay, data fragmentation and resource competition defects of traditional offline analysis solutions.

[0006] To solve the above technical problems, the technical scheme adopted by the present application is as follows: a metadata real-time monitoring method of a distributed storage system, comprising the following steps:

[0007] A metadata eavesdropping service physically isolated from the master node and the standby node is constructed, and the eavesdropping service only receives metadata change streams and is prohibited from participating in node election;

[0008] Dynamic loading of file system image files, parsing and building a memory metadata model containing complete namespace topology;

[0009] Real-time subscription to metadata change event streams published by distributed log nodes, ensuring event timing consistency through transaction ordering mechanisms;

[0010] Incremental update of the memory metadata model, version conflict detection and automatic merging during synchronization;

[0011] Extracting preset key analysis dimension fields in metadata and converting them into structured key-value pair format;

[0012] Batch compression of structured metadata and synchronization to the index engine to establish a dual-index structure;

[0013] Responding to query requests and returning analysis results of cross-level directory topology, user behavior portrait, and storage block distribution balance.

[0014] Further, the dynamic loading of file system image files includes:

[0015] Identifying file paths, permission configurations, storage block distribution, and replica location information;

[0016] Building a tree-shaped memory structure, where the root node corresponds to the storage cluster namespace, the branch node maps the directory level, and the leaf node associates file attributes and physical block distribution.

[0017] Further, the real-time subscription to metadata change event streams published by distributed log nodes includes:

[0018] Capturing EditLog transaction streams published by JournalNode through asynchronous message queues;

[0019] Filtering file creation / deletion, block allocation change, and attribute modification events, and discarding heartbeat detection and node state update events.

[0020] Further, the incremental update of the memory metadata model, version conflict detection and automatic merging during synchronization includes:

[0021] Global timestamp ordering of concurrent events;

[0022] Detecting cross-transaction conflicts of directory movement and file renaming operations;

[0023] When conflicts occur, automatically rolling back and reapplying the latest change transactions based on version numbers.

[0024] Further, the key analysis dimension field includes: path depth, copy number, latest access timestamp, file size, cross-rack storage block distribution identification; and operation frequency and storage quota usage rate are aggregated according to user attribution labels.

[0025] Further, the step of synchronizing the batch-compressed structured metadata to the index engine to establish a double-index structure includes:

[0026] The JSON format metadata is divided into fragments according to a directory level;

[0027] The fragment data is compressed by using dictionary encoding;

[0028] A path hash index supporting prefix matching query and a multi-dimensional label inverted index supporting aggregation analysis are established.

[0029] Further, the analysis result includes:

[0030] A file quantity and storage capacity heat map counted according to a directory level;

[0031] Trend prediction of operation frequency and quota usage rate under a user dimension;

[0032] Copy imbalance risk warning of a storage block across nodes / racks.

[0033] The application further provides a metadata real-time monitoring device of a distributed storage system, which comprises:

[0034] A side listening service module is configured to build a metadata side listening service physically isolated from master nodes and standby nodes, wherein the side listening service only receives metadata change streams and is prohibited from participating in node election;

[0035] A mirror analysis module is configured to dynamically load a file system mirror file, analyze and build a memory metadata model containing a complete namespace topology;

[0036] An event stream subscription module is configured to subscribe to metadata change event streams published by distributed log nodes in real time, and ensure event time sequence consistency through a transaction ordering mechanism;

[0037] An incremental synchronization module is configured to incrementally update the memory metadata model, and perform version conflict detection and automatic merging during synchronization;

[0038] A field extraction module is configured to extract preset key analysis dimension fields in metadata and convert them into a structured key-value pair format;

[0039] An index establishment module is configured to synchronize batch-compressed structured metadata to an index engine to establish a double-index structure;

[0040] An analysis engine module is configured to respond to a query request and return analysis results of cross-hierarchy directory topology, user behavior profiling, and storage block distribution balance.

[0041] The application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the distributed file system metadata monitoring method when executing the computer program.

[0042] The application further provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the distributed file system metadata monitoring method.

[0043] The application has the beneficial effects that: the isolated eavesdropping service is constructed to realize zero-performance-loss real-time metadata monitoring; the file system image file is dynamically loaded and the incremental update model is used to improve the metadata collection timeliness from the hour level to the second level, and the large-scale cluster scanning is accelerated by 100 times; the structured index engine supports millisecond-level response directory topology analysis, user behavior profiling and replica distribution detection, and actively identifies deep redundant directories and cross-rack replica imbalance; the eavesdropping mechanism avoids the I / O competition of the master node, guarantees the production business delay, retains the full metadata fields such as the extension attribute, and solves the data fragmentation defect of the traditional solution. BRIEF DESCRIPTION OF DRAWINGS

[0044] The specific structure of the application will be described in detail below with reference to the drawings.

[0045] Figure 1 The metadata real-time monitoring method flowchart of the distributed storage system of the embodiment of the application;

[0046] Figure 2 The mirror image analysis flowchart of the embodiment of the application;

[0047] Figure 3 The event stream subscription flowchart of the embodiment of the application;

[0048] Figure 4 The incremental synchronization flowchart of the embodiment of the application;

[0049] Figure 5 The index establishment flowchart of the embodiment of the application;

[0050] Figure 6 The metadata real-time monitoring device block diagram of the distributed storage system of the embodiment of the application;

[0051] Figure 7 The schematic block diagram of the computer device of the embodiment of the application. DETAILED DESCRIPTION

[0052] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of the present application.

[0053] It should be understood that the terms "comprise" and "include" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0054] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should be further understood that the term "and / or" as used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0056] As shown in Figure 1 The embodiments of the present application are as follows: a metadata real-time monitoring method of a distributed storage system, comprising the following steps:

[0057] S10, a metadata eavesdropping service physically isolated from the master node and the standby node is constructed, and the eavesdropping service only receives a metadata change stream and is prohibited from participating in node election;

[0058] In the present embodiment, the master node (Active NameNode) and the standby node (Standby NameNode) are deployed in an independent server cluster. The metadata eavesdropping service (NNS) is deployed in a dedicated hardware resource pool, and physical isolation is achieved through network isolation strategies: a dedicated network switch is configured to limit the NNS to only receive a one-way data stream from the log node cluster; the communication ability of the election protocol port is disabled to ensure that the NNS cannot respond to the master-standby switching signal. This scheme can avoid resource contention of the master node caused by monitoring tasks, and prevent system shock caused by mis-triggering of cluster election.

[0059] S20, a file system image file is dynamically loaded, and a memory metadata model containing a complete namespace topology is parsed and constructed;

[0060] As shown in Figure 2As shown, in one specific embodiment, step S20, the dynamic loading of the file system image file includes:

[0061] S21. Identify file path, permission configuration, storage block distribution, and copy location information;

[0062] S22. Construct a tree-shaped memory structure, where the root node corresponds to the storage cluster namespace, branch nodes map to directory levels, and leaf nodes are associated with file attributes and physical block distribution.

[0063] In this embodiment, after the file system image (FsImage) is loaded, the parsing process adopts a hierarchical mapping mechanism: the file path is split into multiple segments according to the directory hierarchy, and each segment generates a unique hash identifier; the storage block distribution information is associated with the physical node topology, recording the rack location of the replica in a tree structure: the root node serves as the namespace entry point, branch nodes store directory hierarchy relationships (e.g., / data / → / user / → / project / ), and leaf nodes mount file attribute sets, including access timestamps, block sizes, etc. This scheme can compress path storage space through hash identifiers, and the tree topology supports fast location of deep files.

[0064] S30: Subscribe to the metadata change event stream published by the distributed log nodes in real time, and ensure the consistency of event sequence through a transaction sorting mechanism;

[0065] like Figure 3 As shown, in a specific embodiment, step S30, the real-time subscription to the metadata change event stream published by the distributed log node includes:

[0066] S31. Capture the EditLog transaction stream published by JournalNode through an asynchronous message queue;

[0067] S32, filter file creation / deletion, block allocation change, attribute modification events, discard heartbeat detection and node status update events.

[0068] In this embodiment, the metadata change event stream of the distributed log node (JournalNode) is captured using a publish-subscribe pattern: a subscription channel is established using a high-throughput message middleware, and events are grouped and transmitted according to transaction sequences; the event filter actively discards auxiliary operations unrelated to metadata changes (such as node state updates). Timing consistency is guaranteed: transactions across log nodes are reorganized according to the global commit order to ensure that the event processing order is consistent with that of the master node. This solution can eliminate event out-of-order delivery caused by network latency and ensure accurate synchronization of the memory model state.

[0069] S40. Incrementally update the memory metadata model, and perform version conflict detection and automatic merging during the synchronization process;

[0070] likeFigure 4 As shown, in a specific embodiment, step S40, the incremental update of the memory metadata model, includes version conflict detection and automatic merging during the synchronization process, comprising:

[0071] S41. Globally sort concurrent events by timestamp;

[0072] S42. Detect cross-transaction conflicts between directory movement and file renaming operations;

[0073] S43. When a conflict occurs, automatically roll back and reapply the latest changes based on the version number.

[0074] In this embodiment, version conflict detection employs a logical clock marking mechanism: each metadata change event carries a globally incrementing version tag; when concurrent operations on the same path are detected (such as file renaming and directory moving), an automatic merging strategy is used to compare the version tag timings: the earlier operation is automatically rolled back, the latest version change is reapplied, and the conflict resolution status is marked in the memory model. This solution can resolve data inconsistency issues caused by concurrent operations in a distributed environment, ensuring strong consistency of the memory model.

[0075] S50. Extract the preset key analysis dimension fields from the metadata and convert them into structured key-value pair format;

[0076] In one specific embodiment, the key analysis dimension fields include: path depth, number of replicas, most recently accessed timestamp, file size, cross-rack storage block distribution identifier; and aggregated operation frequency and storage quota utilization rate by user affiliation tag.

[0077] In this embodiment, key analysis dimension fields include: cross-rack storage identifier: generating hierarchical tags based on the number of physical racks where the replicas are distributed; user behavior profile field: aggregating user historical operation frequency and storage quota usage ratio. Structured transformation: the original metadata is reorganized into a nested key-value structure, where the path field is decomposed into parent-child relationships hierarchically, and user tags serve as independent analysis dimensions. This solution can transform unstructured metadata into a semi-structured data format that supports aggregate analysis.

[0078] S60. After batch compression of structured metadata, synchronize it to the index engine to establish a dual-index structure;

[0079] like Figure 5 As shown, in a specific embodiment, step S60, which involves batch compressing the structured metadata and synchronizing it to the index engine to establish a dual-index structure, includes:

[0080] S61. Slice the JSON format metadata according to the directory hierarchy;

[0081] S62. Use dictionary encoding to compress fragmented data;

[0082] S63, establish a path hash index supporting nested prefix matching query and a multi-dimensional label inverted index supporting aggregation analysis.

[0083] In this embodiment, the batch compression process includes:

[0084] Data fragmentation: divide the metadata subset according to the directory hierarchy to avoid data overload in single transmission;

[0085] Dictionary encoding: generate a fixed-length code for high-frequency path prefixes to significantly reduce storage overhead;

[0086] Dual index mechanism: path index supports prefix-based recursive query, such as retrieving all sub-files under / data / user / ;

[0087] Label inverted index supports multi-dimensional combined filtering, such as "user A and quota usage higher than threshold"; this scheme can compress transmission to improve synchronization efficiency, and the dual index structure takes into account both accurate matching and complex analysis scenarios.

[0088] S70, respond to query requests and return analysis results of cross-hierarchy directory topology, user behavior portrait, and storage block distribution balance.

[0089] In a specific embodiment, the analysis results include:

[0090] File quantity and storage capacity heat map counted according to directory hierarchy;

[0091] Trend prediction of operation frequency and quota usage under user dimension;

[0092] Copy imbalance risk warning of storage blocks across nodes / racks.

[0093] In this embodiment, the analysis results include:

[0094] Directory heat map: visualize file density at each level through color depth gradient to highlight deep nesting areas;

[0095] User behavior trend: predict storage quota depletion time point based on historical operation sequence;

[0096] Copy imbalance warning: when detecting that copies are concentrated in a small number of physical racks, generate a risk prompt.

[0097] This scheme can convert raw data into actionable operational insights to guide storage optimization and resource scheduling.

[0098] In summary, the embodiment of the application captures real-time data stream through physically isolated eavesdropping service, solves concurrency conflicts in combination with logical clock, constructs a memory metadata model completely synchronized with production environment, adopts dictionary encoding compression and hierarchical indexing mechanism to realize efficient storage and millisecond-level retrieval of massive metadata, and finally converts raw data into directory topology portrait, user behavior prediction and storage risk early warning through a multi-dimensional analysis model, thereby significantly improving observability and operation and maintenance efficiency of the distributed file system.

[0099] As shown in Figure 6 The embodiment of the application also provides a metadata real-time monitoring device of a distributed storage system, which comprises:

[0100] The eavesdropping service module 10 is configured to construct a metadata eavesdropping service physically isolated from the master node and the standby node, wherein the eavesdropping service only receives metadata change stream and is prohibited from participating in node election.

[0101] The mirror analysis module 20 is configured to dynamically load a file system mirror file, analyze and construct a memory metadata model containing complete namespace topology.

[0102] The event stream subscription module 30 is configured to subscribe to a metadata change event stream published by a distributed log node in real time, and ensure event time sequence consistency through a transaction ordering mechanism.

[0103] The incremental synchronization module 40 is configured to incrementally update the memory metadata model, and perform version conflict detection and automatic merging during synchronization.

[0104] The field extraction module 50 is configured to extract preset key analysis dimension fields in the metadata and convert them into a structured key-value pair format.

[0105] The index establishment module 60 is configured to synchronize the structured metadata after batch compression to an index engine and establish a double-index structure.

[0106] The analysis engine module 70 is configured to respond to a query request and return analysis results of cross-level directory topology, user behavior portrait and storage block distribution balance.

[0107] In a specific embodiment, the mirror analysis module 20 is specifically configured to:

[0108] Identify file path, permission configuration, storage block distribution and replica location information.

[0109] Construct a tree-shaped memory structure, wherein a root node corresponds to a storage cluster namespace, a branch node maps a directory level, and a leaf node is associated with file attributes and physical block distribution.

[0110] In a specific embodiment, the event stream subscription module 30 is specifically configured to:

[0111] Capture EditLog transaction stream published by JournalNode through asynchronous message queue;

[0112] Filter file create / delete, block allocation change, property modification events, discard heartbeat detection and node state update events.

[0113] In an embodiment, the incremental synchronization module 40 is specifically configured to:

[0114] Sort concurrent events globally by timestamp;

[0115] Detect cross-transaction conflicts of directory move and file rename operations;

[0116] When a conflict occurs, automatically roll back and reapply the latest change transaction based on version number.

[0117] In an embodiment, the index building module 60 is specifically configured to:

[0118] Shard JSON format metadata by directory hierarchy;

[0119] Dictionary encoding compression is adopted for shard data;

[0120] Build path hash index supporting prefix matching query and multi-dimensional label inverted index supporting aggregation analysis.

[0121] It should be noted that the specific implementation process of the above-mentioned distributed file system metadata monitoring device and each unit can be clearly understood by those skilled in the art, which can be referred to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0122] The above-mentioned distributed file system metadata monitoring device can be realized in the form of a computer program, which can run on a computer device as shown in Figure 7 .

[0123] Please refer to Figure 7 , Figure 7 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a terminal or a server, wherein the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant, a wearable device, and the like electronic device having a communication function. The server can be a stand-alone server or a server cluster composed of multiple servers.

[0124] Please refer to Figure 7The computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0125] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions which, when executed, can cause the processor 502 to perform a distributed file system metadata monitoring method.

[0126] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0127] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, which, when executed by the processor 502, can cause the processor 502 to perform a distributed file system metadata monitoring method.

[0128] The network interface 505 is configured to perform network communication with other devices. Those skilled in the art can understand that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement. ​

[0129] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the distributed file system metadata monitoring method as described above.

[0130] It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0131] ​Those skilled in the art can understand that all or part of the processes in the method of the above embodiments can be completed by instructing the relevant hardware by a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above method embodiments.

[0132] Therefore, the application further provides a storage medium. The storage medium can be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions are executed by a processor to make the processor execute the distributed file system metadata monitoring method as described above.

[0133] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various computer readable storage media that can store program codes.

[0134] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0135] In several embodiments provided by the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be omitted or not executed.

[0136] The steps in the method of the embodiments of the application can be adjusted, combined and deleted in sequence according to actual needs. The units in the device of the embodiments of the application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0137] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a storage medium. Based on such an understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application.

[0138] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A metadata real-time monitoring method of a distributed storage system, characterized in that, The method comprises the following steps: Constructing a metadata eavesdropping service physically isolated from the master node and the standby node, which only receives the metadata change stream and is prohibited from participating in node election; Dynamically loading the file system image file, parsing and constructing a memory metadata model containing a complete namespace topology; Real-time subscription to the metadata change event stream published by the distributed log node, ensuring event timing consistency through transaction sequencing mechanism; Incrementally updating the memory metadata model, performing version conflict detection and automatic merging during synchronization process; Extracting preset key analysis dimension fields in the metadata and converting them into structured key-value pair format; Synchronizing the structured metadata to the index engine after batch compression, establishing a double index structure; Responding to query requests and returning analysis results of cross-level directory topology, user behavior portrait and storage block distribution balance.

2. The metadata real-time monitoring method of the distributed storage system according to claim 1, characterized in that, The dynamically loading the file system image file comprises: Identifying file path, permission configuration, storage block distribution and replica location information; Constructing a tree-shaped memory structure, where the root node corresponds to the storage cluster namespace, the branch node maps the directory level, and the leaf node is associated with file attributes and physical block distribution.

3. The method of claim 1, wherein, The real-time subscription to the metadata change event stream published by the distributed log node comprises: Capturing the EditLog transaction stream published by the JournalNode through an asynchronous message queue; Filtering file creation / deletion, block allocation change, attribute modification events, and discarding heartbeat detection and node state update events.

4. The metadata real-time monitoring method of the distributed storage system according to claim 1, characterized in that, The incrementally updating the memory metadata model, performing version conflict detection and automatic merging during synchronization process comprises: Global sorting of concurrent events by timestamp; Detecting cross-transaction conflicts of directory movement and file renaming operations; When a conflict occurs, automatically rolling back and reapplying the latest change transaction based on version number.

5. The metadata real-time monitoring method of the distributed storage system according to claim 1, characterized in that, The key analysis dimension fields include: path depth, number of replicas, latest access timestamp, file size, cross-rack storage block distribution identifier; aggregating operation frequency and storage quota usage rate according to user attribution label.

6. The method of claim 1, wherein, The synchronizing the structured metadata to the index engine after batch compression, establishing a double index structure comprises: Sharding JSON format metadata by directory level; Dictionary encoding compression of sharded data; Establishing path hash index supporting prefix match query and multi-dimensional label inverted index supporting aggregation analysis.

7. The method of claim 1, wherein, The analysis results include: File quantity and storage capacity heat map counted by directory level; Trend prediction of operation frequency and quota usage rate under user dimension; Copy imbalance risk warning of storage blocks across nodes / racks.

8. A metadata real-time monitoring apparatus of a distributed storage system, characterized by comprising: It comprises: An eavesdropping service module for constructing a metadata eavesdropping service physically isolated from the master node and the standby node, which only receives the metadata change stream and is prohibited from participating in node election; An image parsing module for dynamically loading the file system image file, parsing and constructing a memory metadata model containing a complete namespace topology; An event stream subscription module for real-time subscription to the metadata change event stream published by the distributed log node, ensuring event timing consistency through transaction sequencing mechanism; An incremental synchronization module for incrementally updating the memory metadata model, performing version conflict detection and automatic merging during synchronization process; An A field extraction module is configured to extract a preset key analysis dimension field in the metadata and convert the key analysis dimension field into a structured key-value pair format; An index establishment module is configured to synchronize the batch compressed structured metadata to an index engine and establish a double index structure; An analysis engine module is configured to respond to a query request and return an analysis result of a cross-level directory topology, a user behavior portrait and a storage block distribution balance.

9. A computer device, comprising: The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the distributed file system metadata monitoring method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium characterized by: The storage medium stores a computer program, and the computer program can implement the distributed file system metadata monitoring method according to any one of claims 1 to 7 when executed by a processor.

Images