Metadata processing method based on data snapshot and electronic device

Abstract

This application discloses a metadata processing method and electronic device based on data snapshots, relating to the field of data processing. It establishes a metadata generation benchmark through snapshot identifiers, divides generations by time increments and embeds them into metadata keys, synchronizes the generation benchmarks of multiple controllers to ensure consistency, forces a flush of the previous generation's cached data to complete a full summary, and then constructs targeted filters to match the metadata of the previous generation. This clearly defines the metadata generation attribution, solving the problem of inaccurate metadata validity judgment caused by inaccurate generation benchmarks in multi-controller storage systems, thereby improving judgment accuracy and storage resource reclamation efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a metadata processing method and electronic device based on data snapshots. Background Technology

[0002] Metadata, as the core data of a storage system, records crucial information such as data storage location, attributes, and access permissions. Its validity directly determines the accuracy of data reclamation and storage resource processing, making it a core technology in the field of storage system data management. In related technologies, metadata processing typically involves receiving read / write requests from the host, classifying and storing the metadata associated with the requests, and verifying the validity of the metadata in the storage medium during garbage collection to determine whether to retain or release the data. This entire process relies on a controller cluster to complete parallel data processing and interaction.

[0003] In a storage system composed of multiple controllers, metadata processing requires a unified generation benchmark for orderly management. However, the related technologies lack a precise synchronization mechanism for the metadata generation benchmark during the collaboration between controllers, resulting in confusion about the generation of metadata processed by each controller. This leads to omissions or misjudgments in subsequent metadata validity assessments, making it impossible to efficiently and accurately reclaim and optimize storage resources. Summary of the Invention

[0004] This application provides a metadata processing method and electronic device based on data snapshots, in order to at least solve the problems of time-consuming and inaccurate metadata validity judgment in related technologies.

[0005] This application provides a metadata processing method based on data snapshots, including:

[0006] Obtain the metadata validity judgment requirements of the storage system and the request data of the host. The request data is the metadata to be processed corresponding to the write input / output request issued by the host.

[0007] A timer is started based on the validity judgment requirement to generate an initial snapshot identifier and increment it at preset time intervals to obtain the current snapshot identifier.

[0008] The current snapshot identifier is embedded in the key of the request data to obtain the target metadata with time generation mark and stored in the database for persistent processing;

[0009] Synchronize the current snapshot identifiers among multiple controllers of the storage system;

[0010] Force flush the cached target metadata that has not been flushed and corresponds to the previous generation snapshot identifier to obtain generational metadata;

[0011] A validity judgment filter corresponding to the previous generation snapshot identifier is constructed based on the generational metadata.

[0012] The validity filter matches the metadata corresponding to the previous generation snapshot identifier in the database to obtain the metadata processing result, and returns it to the timer for the next generation processing until the preset termination condition is met. The metadata processing result includes retaining the previous generation metadata and storing it in a new location or destroying it.

[0013] This application also provides a metadata processing apparatus based on data snapshots, comprising:

[0014] The acquisition module is used to acquire the metadata validity judgment requirements of the storage system and the request data of the host. The request data is the metadata to be processed corresponding to the write input / output request issued by the host.

[0015] The identification module is used to start a timer based on the validity judgment requirement, generate an initial snapshot identifier, and increment it sequentially at preset time intervals to obtain the current snapshot identifier;

[0016] The embedding module is used to embed the current snapshot identifier into the key of the request data, obtain the target metadata with time generation mark, and store it in the database for persistent processing;

[0017] The synchronization module is used to synchronize the current snapshot identifiers among multiple controllers of the storage system;

[0018] The flush module is used to perform a forced flush operation on the target metadata in the cache that corresponds to the previous generation snapshot identifier and has not been flushed, so as to obtain the generational metadata;

[0019] The building module is used to construct a validity judgment filter corresponding to the previous generation snapshot identifier based on the generational metadata;

[0020] The processing module is used to match the metadata corresponding to the previous generation snapshot identifier in the database through a validity judgment filter, obtain the metadata processing result, and return it to the timer for the next generation processing until the preset termination condition is met. The metadata processing result includes retaining the previous generation metadata and storing it in a new location or destroying it.

[0021] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described metadata processing methods based on data snapshots when executing the computer program.

[0022] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described metadata processing methods based on data snapshots.

[0023] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described metadata processing methods based on data snapshots.

[0024] By using snapshot identifiers to construct a metadata generation division and synchronization system, synchronizing the generation benchmarks of multiple controllers to ensure coordination and consistency, and completely reclaiming the cached data of the previous generation to clarify the processing boundaries, the problem of ambiguous metadata generation ownership is eliminated from the root. Therefore, the technical problem of inaccurate metadata validity judgment caused by inaccurate generation benchmarks in multi-controller storage systems can be solved, thereby achieving the technical effect of improving judgment accuracy and optimizing storage resource reclamation efficiency. Attached Figure Description

[0025] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart illustrating the metadata processing method based on data snapshots provided in this application;

[0027] Figure 2 The flowchart provided for this application;

[0028] Figure 3 A schematic diagram of the structure of the metadata processing device based on data snapshot provided in this application;

[0029] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, other embodiments obtained by those of ordinary skill in the art without creative effort are all within the protection scope of this application.

[0031] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0032] This application uses snapshot identifiers as the basis for metadata generation, and combines timers to achieve periodic incrementing of generations. The snapshot identifier is embedded in the metadata key-value pair to achieve a unique binding between metadata and generation. Synchronization of snapshot identifiers among multiple controllers ensures consistency in generational benchmarks across controllers. Then, a forced flush of the previous generation's metadata from the cache is performed to complete the full aggregation. Based on the aggregated generational metadata, a dedicated validity filter is constructed to accurately match the previous generation's metadata in the database to determine validity, forming a generation-by-generation cyclical processing mechanism to ensure the orderly and accurate processing of metadata.

[0033] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Figure 1 A flowchart illustrating the metadata processing method based on data snapshots provided in this application; Figure 2 The operation flowchart provided for this application is as follows: Figure 1 and Figure 2 As shown, this embodiment provides a metadata processing method based on data snapshots, which includes the following steps:

[0035] Step S101: Obtain the metadata validity judgment requirements of the storage system and the request data of the host. The request data is the metadata to be processed corresponding to the write input / output request issued by the host.

[0036] Specifically, metadata refers to data used to describe core information such as data storage location, attributes, and access rules; request data refers to the information associated with the original data that needs to be processed and stored when the host sends a write input / output request to the storage system.

[0037] Based on the communication protocol between the storage system and the host, the system's core task requirements (metadata validity judgment) and the data to be processed on the host side are captured in real time through the data interface, establishing a corresponding relationship between requirements and data. By clarifying the goal of this data processing (response validity judgment requirement) and the processing object (the metadata to be processed corresponding to the host request data), core inputs are provided for subsequent generation division, identifier binding, and other operations, avoiding the disconnect between data processing and system requirements.

[0038] Step S102: Start a timer based on the validity judgment requirement, generate an initial snapshot identifier, and increment it at preset time intervals to obtain the current snapshot identifier.

[0039] Specifically, a snapshot identifier is a sequence of characters or numbers used to uniquely identify the time period to which metadata belongs, and its increment logic follows preset numerical or encoding rules. Through the periodic triggering mechanism of a timer, the initial snapshot identifier is used as a baseline, and an incrementing operation is performed each time it is triggered, forming a continuous sequence of generation identifiers, thus establishing the basis for the time-dimensional metadata division.

[0040] By constructing a core benchmark for metadata generation, and through periodic increments, metadata is systematically categorized by time, resolving the processing chaos caused by mixed metadata from multiple generations, and providing clear generational boundaries for subsequent generational processing. The generational attribution of metadata can be directly defined through snapshot identifiers, achieving automation and standardization of generational division, and providing a unified generational reference for operations such as controller synchronization and data refresh.

[0041] Step S103: Embed the current snapshot identifier into the key of the request data to obtain the target metadata with time generation mark and store it in the database for persistence processing.

[0042] Specifically, target metadata refers to structured metadata bound to snapshot identifiers (generation information). By parsing the field structure of the requested data key, the current snapshot identifier is converted according to the storage system encoding standard and embedded into the specified field of the key, forming a binding relationship between the data content and the generation identifier. Then, the database's persistence engine writes it to the storage medium to ensure that the data is not lost in the event of power failure.

[0043] By binding metadata with its generation, the time attribution of each piece of metadata can be directly identified through key information. At the same time, persistent processing ensures the stability of data storage and avoids subsequent processing interruptions caused by the loss of generational metadata due to temporary storage. It provides identifiable generational tags for the accurate screening and matching of metadata of the previous generation, while solving the problem of easy loss of temporary metadata storage and ensuring the continuity of the data processing chain.

[0044] Step S104: Synchronize the current snapshot identifiers among multiple controllers of the storage system.

[0045] Specifically, based on the distributed cluster communication protocol, a master-slave collaborative synchronization mode is adopted. The preset controller serves as the reference node, which distributes the current snapshot identifier and validity verification information to other controllers through the internal communication link. Each controller updates its local storage after confirming the integrity and accuracy of the identifier through the verification mechanism.

[0046] By unifying the generation benchmark of multiple controllers, the deviation in metadata generation judgment caused by inconsistent snapshot identifiers of each controller is avoided, ensuring that the controllers process metadata in the same time dimension, thus solving the core problem of generation benchmark chaos in a distributed environment; achieving collaborative consistency of generation information of multiple controllers, providing a unified reference for cross-controller metadata flushing, aggregation and matching, and ensuring the consistency and accuracy of distributed processing.

[0047] Step S105: Perform a forced flush operation on the target metadata in the cache that corresponds to the previous generation snapshot identifier and has not been flushed, to obtain generational metadata.

[0048] Specifically, generational metadata refers to a complete metadata dataset belonging to the same previous generation and formed after aggregation. By locating the non-persistent data associated with the previous generation snapshot identifier, a parallel write algorithm is used to write the data in blocks to a specified partition of the database. Combined with the database's write status feedback mechanism, the data storage is confirmed to be complete, and finally, the complete dataset is formed by aggregation according to the generational dimension.

[0049] This ensures that all metadata from the previous generation is migrated from the cache to persistent storage without any omissions, avoiding incomplete metadata due to cache data loss. This provides a comprehensive and accurate data source for subsequent filter construction. It also eliminates data differences between the cache and the database, forming a complete set of metadata from the previous generation. This solves the problem of filter construction deviation caused by incomplete data, providing reliable data support for validity judgment.

[0050] Step S106: Construct a validity judgment filter corresponding to the previous generation snapshot identifier based on the generational metadata.

[0051] Specifically, the validity judgment filter refers to a bitmap structure built on a hash mapping mechanism for quickly determining the validity of metadata. It extracts the location identifier and attribute features of the subdivided metadata, uses a preset hash algorithm to convert the features into hash values, merges them through bit operations, and maps them to a bitmap of a preset length. Valid data features are then marked through bit-setting operations, forming a dedicated judgment model.

[0052] By establishing the criteria for judging the validity of previous generation metadata, the complex metadata comparison is transformed into a fast query of bitmap positions, solving the problem of low efficiency in traditional line-by-line comparison. At the same time, the generation-specific filter ensures the accuracy of the judgment range, effectively improving the efficiency of metadata validity judgment. Furthermore, feature mapping ensures the accuracy of the judgment, avoiding misjudgments caused by cross-generation data interference, and providing an efficient tool for subsequent matching operations.

[0053] Step S107: Match the metadata corresponding to the previous generation snapshot identifier in the database through the validity judgment filter to obtain the metadata processing result, and return it to the timer to process the next generation until the preset termination condition is met. The metadata processing result includes retaining the previous generation metadata and storing it in a new location or destroying it.

[0054] Specifically, the matching logic based on feature consistency extracts the core features of the previous generation of sub-data in the database and converts them into hash values ​​to be matched. By locating the corresponding position of the filter bitmap, the validity of the data is judged by the numerical status. At the same time, the periodic triggering of the timer forms a closed loop of processing and iteration until the termination condition is triggered.

[0055] It effectively completes the validity determination and processing (retention or destruction) of the previous generation of metadata, and at the same time, it realizes the sequential processing of metadata of each generation through a cyclic mechanism, ensuring that all metadata to be processed in the storage system can be overwritten and avoiding generational omissions. It realizes the automation and accuracy of metadata validity judgment, and ensures the long-term stable operation of the storage system through cyclic processing. It solves the problems of incomplete generational coverage and low judgment efficiency in traditional processing, and optimizes storage resource utilization.

[0056] The metadata processing method based on data snapshots provided in this invention establishes a metadata generation division and synchronization system with snapshot identifiers as the core, binds metadata with generations and ensures consistency of multi-controller benchmarks, combines forced flushing to ensure generation data integrity, and implements a precise judgment and cyclic processing mechanism through dedicated filters. This achieves orderly, accurate, and efficient metadata processing, effectively solving problems such as chaotic generation benchmarks, incomplete data processing, and validity judgment deviations in multi-controller storage systems, and significantly improving storage resource reclamation efficiency and system operation stability.

[0057] This embodiment provides a detailed description of the process described in the above embodiment, which involves starting a timer based on validity judgment requirements, generating an initial snapshot identifier, and incrementally incrementing it at preset time intervals to obtain the current snapshot identifier. The specific implementation of this process includes the following steps:

[0058] Step a1: Set the initial snapshot identifier to a preset baseline value and store it in the database.

[0059] Specifically, the initial snapshot identifier is the initial reference identifier used to initiate the generation division of metadata. Its preset baseline value follows the metadata encoding specifications preset by the storage system. The preset baseline value is written into the metadata-specific database through the persistent storage interface of the database to form the initial data record for generation division, ensuring the stability and traceability of the baseline value.

[0060] By establishing a starting point for metadata generation, we can avoid generation sequence chaos caused by missing or inconsistent initial identifiers. At the same time, the database storage ensures that the initial benchmark value will not be lost due to system restarts or temporary failures, providing a reliable starting basis for subsequent generation increments. This gives metadata generation a clear and stable starting point, solving the problems of unfounded and easily lost generation benchmarks, and ensuring the continuity and consistency of the generation sequence.

[0061] Step a2: Monitor the trigger status of the timer. When the timer reaches the preset time interval, increment the current snapshot identifier.

[0062] Specifically, a timer is a timing component used to control the increment cycle of generations. Its triggering status is monitored in real time by the system clock signal. When the accumulated time reaches the preset time interval, the increment logic of the snapshot identifier is triggered. The increment operation follows the preset ordered encoding rules, such as numerical increment, character sequence iteration, etc., to ensure that the incremented identifier is unique and sortable.

[0063] By periodically triggering generational division, metadata is categorized in an orderly manner according to a fixed time dimension, avoiding the randomness and disorder of generational division. At the same time, through clear time interval control, the efficiency of metadata processing and system resource consumption are balanced, forming a continuous and orderly sequence of generational identifiers. This solves the problem of chaotic metadata generational attribution caused by irregular generational updates, ensuring that metadata at different time stages can be accurately distinguished by generation, and providing clear time boundaries for subsequent generational processing.

[0064] Step a3: Update the incremented snapshot identifier to the current snapshot identifier.

[0065] Specifically, the current snapshot identifier refers to the generation identifier bound to the metadata at the current time stage. Through the data synchronization mechanism, the incremented new identifier is synchronously updated to the local controller cache and the metadata-specific database to ensure the consistency of the identifier between local storage and persistent storage, while overwriting the old current snapshot identifier and completing the iteration of the generation benchmark.

[0066] By updating the current benchmark for generation division, subsequent received metadata can be bound to the latest generation identifier, avoiding misbinding of new metadata with old generation identifiers. At the same time, it ensures the benchmark consistency for subsequent synchronization among multiple controllers, connecting the processing of the previous generation with the reception of data in the current generation. It realizes real-time iteration and consistency maintenance of the generation benchmark, solves the problem of metadata generation confusion caused by inconsistent benchmarks after generation updates, ensures that newly received metadata is accurately assigned to the current generation, and provides a guarantee for the parallel execution of subsequent processing of data from the previous generation and storage of data in the current generation.

[0067] This invention establishes the generation starting point by setting a preset benchmark value and persistently storing it. It forms an ordered generation sequence by periodically triggering the snapshot identifier to increment through a timer and synchronously updating the current snapshot identifier to ensure benchmark consistency. This achieves standardization, automation, and continuity of metadata generation division, effectively solving the problems of generation benchmark lack of basis, irregular updates, and inconsistent benchmarks. It provides a precise and stable generation management foundation for multi-controller collaboration and generation-by-generation metadata processing.

[0068] This embodiment provides a detailed description of the process of embedding the current snapshot identifier into the key of the request data in the above embodiments. The specific implementation of this process includes the following steps:

[0069] Step b1: Parse the key structure of the request data to obtain the identifier embedding field used to carry time generation information.

[0070] Specifically, the key structure of the requested data refers to a structured data format containing multiple functional fields used to uniquely identify the requested data. Based on the storage system's preset key definition specifications, the key structure parsing engine traverses the field composition of the key, identifies and extracts the preset identifier embedded field specifically used to carry time-series information, and simultaneously verifies the integrity and availability of the fields during the parsing process.

[0071] By clearly defining the exclusive embedding location of the snapshot identifier, the data identifier chaos caused by the embedding of the identifier occupying other functional fields is avoided. At the same time, field validation ensures that the embedding location has effective carrying capacity, providing a structural foundation for the accurate binding of the snapshot identifier and the request data key. This solves the problem of unclear snapshot identifier embedding location, ensures that the embedding operation will not destroy the original function of the key, and provides a guarantee for the standardization and stability of subsequent identifier embedding, making the storage location of generational information uniform and traceable.

[0072] Step b2: Convert the current snapshot identifier according to the storage system's preset encoding specifications to obtain a standard identifier.

[0073] Specifically, the preset encoding standard refers to the unified data format standard established by the storage system to ensure data compatibility and parsing efficiency; the standard identifier refers to the snapshot identifier format that conforms to the encoding standard and can be directly embedded in the key field. Through the encoding conversion module, the original snapshot identifier is converted into a fixed-length, unified character set format according to the standard, and a format verification is performed synchronously after the conversion to confirm compliance.

[0074] It effectively eliminates the problem of incompatibility between the original snapshot identifier format and the key field format, ensuring that the identifier can be successfully embedded into the field without format conflicts. At the same time, it improves the efficiency and accuracy of subsequent cross-controller and cross-module parsing through unified encoding. It realizes the format adaptation of snapshot identifier and key field, avoids embedding failure or parsing errors caused by format differences, and makes generation information universal across components, providing a unified format basis for subsequent metadata generation identification.

[0075] Step b3: Write the standard identifier into the identifier embedding field to obtain the target metadata with time generation mark.

[0076] Specifically, target metadata refers to metadata that is bound to time-generation information (standard identifier) ​​and has complete generation attributes. Through the field writing interface, the standard identifier is written into the identifier embedding field according to a specified format. After writing, a data validation mechanism confirms the completeness of the binding between the identifier and the field, ensuring that the generation information is associated with the metadata subject.

[0077] This approach achieves a strong association between metadata and its generation, enabling the time generation attribute of metadata to be directly identified through the key structure. This provides an intuitive basis for generation identification in the subsequent filtering, summarization, and validity judgment of metadata from the previous generation. It solves the problem of ambiguous generation attribution caused by the separation of metadata and generation information, making the generation attribute of metadata clear and verifiable, and providing a data foundation for subsequent generational processing.

[0078] This invention, through resolving the key structure to clearly define the exclusive embedding position, converting the snapshot identifier format according to the encoding standard, and accurately binding the standard identifier with the metadata, achieves the standardization, compatibility, and relevance of metadata generation information storage. It effectively solves the problems of chaotic snapshot identifier embedding, incompatible formats, and separation of generation information and metadata, and provides a reliable generation marking foundation for multi-controller generation synchronization, accurate metadata filtering, and generation-by-generation processing.

[0079] This embodiment details the process of performing a forced flush operation on the target metadata in the cache that has not been flushed and corresponds to the previous generation snapshot identifier in the above embodiments. The specific implementation of this process includes the following steps:

[0080] Step c1: Obtain the target metadata that has not been refreshed from the cache, corresponding to the previous generation snapshot identifier.

[0081] Specifically, un-flushd target metadata refers to metadata that has been bound to a generation identifier but is still temporarily stored in the cache and has not been written to the database for persistent storage. Based on the cache index, using the previous generation snapshot identifier as the filtering keyword, the metadata entries in the cache are traversed, and the un-persistent data belonging to the target generation is filtered out by matching the generation identifier, while simultaneously verifying the binding consistency between the data and the identifier.

[0082] Precisely locate the target data range that needs to be forcibly flushed, avoid irrelevant generation data or flushed data from being mixed into the processing flow, ensure that the flushing operation only targets valid data from the previous generation that has not been persisted, lay the foundation for complete data aggregation, solve the problem of ambiguous filtering range of cached data, achieve accurate extraction of data that has not been flushed, reduce the overhead of invalid data processing, and ensure the generation purity and relevance of subsequent flushed data.

[0083] Step c2 involves writing the un-flushd target metadata to the database through forced flushing, and then using a parallel write algorithm to synchronously write the un-flushd target metadata into blocks to the corresponding storage partitions in the database.

[0084] Specifically, forced flushing refers to the mechanism of proactively triggering data writing to the database by bypassing the regular cache refresh strategy; parallel write algorithm refers to the algorithm that splits data according to preset rules and writes it to the storage medium synchronously through multiple write channels; storage partition refers to a dedicated data storage area in the database divided according to generation identifiers. First, a forced flush command triggers the data writing process, then the data is processed in blocks according to the storage system specifications, the corresponding storage partition is located based on the previous generation snapshot identifier, and the data writing is completed through a multi-channel parallel mechanism.

[0085] It effectively overcomes the time limitations of regular cache refresh, ensuring timely persistence of data that has not yet been flushed. By using block-based and parallel writing, it improves data writing efficiency, while generation-based partitioning avoids the mixing of metadata from different generations, reducing the difficulty of data retrieval for subsequent aggregation operations. It solves the problems of incomplete generation-based metadata and low write efficiency caused by cached data stagnation, achieving fast and orderly persistent data storage and ensuring the partitioning regularity and accessibility of data in the database.

[0086] Step c3: Obtain the target metadata write status not yet refreshed from the database, and confirm that the target metadata has been stored.

[0087] Specifically, write status refers to the status information returned by the database indicating whether the data write was successful and complete (such as write completion flags, data verification results, etc.). Through the database's status feedback interface, the write results of each data block are received in real time. Combined with block hash verification and global data volume comparison, the integrity and accuracy of the data write are verified, forming a closed-loop confirmation mechanism.

[0088] To avoid data loss or corruption due to network interruptions or storage anomalies during the data writing process, status feedback and verification ensure that the refreshed data is consistent with the original data, solve the problem of lack of validity verification after data writing, eliminate the hidden danger of incomplete generational metadata caused by write anomalies, and ensure that the previous generation of metadata stored in the database is complete and accurate.

[0089] Step c4: Summarize the target metadata corresponding to the previous generation snapshot identifier stored in the storage partition to obtain generational metadata.

[0090] Specifically, generational metadata refers to a standardized metadata dataset belonging to the same previous generation and formed after complete aggregation. Based on the snapshot identifier of the previous generation, the target metadata within the corresponding storage partition is traversed, and duplicate entries are removed and the data format is standardized through data integration algorithms to form a generation-specific dataset with a unified structure and complete content.

[0091] By integrating individual metadata records scattered across partitions, a generational data set with overall processing value is formed. This provides a comprehensive and well-organized data source for the construction of subsequent validity assessment filters, ensuring that the filters can cover the previous generation of valid data. This avoids the problems of low processing efficiency and incomplete data sources for filter construction caused by the scattered storage of individual metadata records, forming a unified and comprehensive generational metadata set, providing core data support for the accuracy of subsequent validity assessments.

[0092] This invention accurately filters out unflushed data in the cache by using the above-mentioned generational snapshot identifier. It improves data persistence efficiency and regularity by using forced flushing, block parallel writing, and partitioned storage. It ensures data integrity through write status feedback and verification, and integrates partitioned data to form standardized generational metadata. This effectively solves the problems of incomplete generational data flushing, low write efficiency, chaotic storage, and insufficient data reliability, and provides comprehensive, accurate, and regular data source support for the construction of the previous generational dedicated validity judgment filter.

[0093] This embodiment provides a detailed description of the process in the above embodiments for constructing a validity judgment filter corresponding to the previous generation snapshot identifier based on generational metadata. The specific implementation of this process includes the following steps:

[0094] Step d1: Perform integrity verification on the epoch-making metadata and delete invalid or corrupt metadata records.

[0095] Specifically, integrity verification refers to the process of verifying the format standardization, field integrity, and data consistency of metadata through preset data verification rules. Based on the preset verification rules, it checks whether the structure of the metadata conforms to the storage system standards, whether key fields are missing, and whether there are logical conflicts in the data content, and directly removes invalid or corrupt records that fail the verification.

[0096] By filtering noisy data in the generational metadata, invalid data is prevented from entering the subsequent processing flow, which would cause filter construction deviations. This ensures that the data source used to build the filter has basic validity, solves the problem of invalid data mixed in the generational metadata, eliminates the interference of invalid data on filter construction, ensures the purity of the data source for subsequent processing, and reduces the occurrence of filter misjudgments.

[0097] Step d2 involves performing deduplication on the verified generational metadata, retaining only the unique and valid metadata entry.

[0098] Specifically, deduplication refers to the process of identifying and merging duplicate data entries based on the core identification information of metadata (such as location identifiers) through feature comparison algorithms. A temporary index table is established based on the unique identification features of the metadata to record the features of processed entries. Subsequent entries are then matched for features, and only the first unique and valid entry is retained.

[0099] By removing duplicate records from the generational metadata, redundant bitmap settings caused by duplicate data are avoided, reducing the space occupation and construction time of the filter. At the same time, it ensures that the feature mapping of the filter for the same metadata is unique, which solves the problem of duplicate entries in generational metadata, simplifies the data source scale, reduces the storage and computing overhead of the filter, avoids redundant filter judgment caused by duplicate data, and improves construction efficiency.

[0100] Step d3: Extract the core feature information of the metadata entries, which includes location identifiers and attribute features.

[0101] Specifically, core feature information refers to the key data dimensions in the generational metadata that can uniquely represent its identity and attributes; location identifier refers to the unique code that marks the physical storage location of the metadata; attribute features refer to the inherent attribute information of the generational metadata, such as creation time and data type. Through feature extraction algorithms, these two types of core features are accurately extracted from the metadata entries to form a structured feature dataset.

[0102] This paper clarifies the core criteria for the filter to judge the validity of metadata, transforms complex metadata entries into quantifiable and computable feature information, provides standardized input for subsequent hash processing, ensures that the filter can make accurate judgments based on the essential features of metadata, solves the problem of unclear metadata feature extraction, achieves accurate stripping of core metadata information, provides targeted input for subsequent hash mapping, and avoids filter judgment bias caused by improper feature selection.

[0103] Step d4: The location identifier and attribute features are processed by a preset hash algorithm to obtain the first hash value and the second hash value.

[0104] Specifically, a predefined hash algorithm refers to a hash calculation rule (such as the MurmurHash3 algorithm) that is predefined by the storage system and has a low collision rate. The extracted location identifier and attribute features are taken as independent inputs and a one-way mapping calculation is performed using the predefined hash algorithm to transform the two types of features into a first hash value and a second hash value of fixed length and uniform distribution, respectively.

[0105] This method transforms high-dimensional feature information into low-dimensional hash values, reducing the computational complexity of subsequent data processing. Simultaneously, the uniform distribution of hash values ​​reduces the probability of collisions in subsequent bitmap mapping, supporting efficient feature processing. It achieves dimensionality reduction and standardization of core feature information, addressing the problem of low efficiency in high-dimensional feature processing. This provides an efficient and easily processed data format for subsequent bitwise operations, merging, and bitmap mapping, improving overall processing speed.

[0106] Step d5: Merge the first hash value and the second hash value according to the preset bitwise operation rules to obtain the merged hash value.

[0107] Specifically, the preset bitwise operation rules refer to the bitwise operation logic (such as bitwise XOR and bitwise AND operations) defined by the storage system for fusing multiple hash values. According to the preset bitwise operation rules, the binary bits of the first hash value and the second hash value are processed to fuse the hash information corresponding to the two types of features into a unified fused hash value, retaining the core distinguishing information of the two types of features.

[0108] By integrating the hash information of location identifiers and attribute features, a comprehensive feature mapping is formed that can simultaneously represent the location and attributes of metadata. This avoids the bias caused by a single feature hash value and improves the comprehensiveness of the filter's judgment on the validity of metadata. It solves the problem that a single feature hash value cannot fully represent metadata by achieving information fusion of two types of core features. This allows the fused hash value to more comprehensively reflect the essence of metadata, providing a more reliable basis for the filter's accurate judgment.

[0109] Step d6: Map the fused hash value to a bitmap of a preset length, set the values ​​at the mapped positions, and generate a validity judgment filter.

[0110] Specifically, a validity filter refers to a probabilistic data structure built on a bitmap structure for quickly retrieving the validity of metadata, such as a Bloom filter. A bitmap is an array of binary bits, where each bit corresponds to a hash value mapping position. This step performs a modulo operation on the fused hash value according to the preset bitmap length to determine its unique mapping position in the bitmap, and sets the binary bit at that position from 0 to 1. After the fused hash value mapping is completed, a complete filter is formed.

[0111] By mapping the core features of metadata to binary bit states in a bitmap, a tool is built that can quickly determine whether metadata belongs to a valid set, enabling rapid retrieval of subsequent metadata validity judgments. This replaces the inefficient traditional method of comparing each entry one by one, solving the problem of low efficiency in traditional metadata validity judgments. A filter with high retrieval speed and low space consumption is constructed, allowing subsequent metadata matching operations to be completed quickly through bitmap bit states, significantly improving judgment efficiency.

[0112] This invention ensures data source quality by performing integrity verification and deduplication on generational metadata, extracts location identifiers and attribute features as core judgment criteria, and forms a comprehensive feature map through hash and bit operations. Finally, a bitmap-based validity judgment filter is constructed, which effectively solves the problems of poor quality, unclear features, low filter construction efficiency, and inaccurate judgment of generational metadata. It achieves high efficiency in filter construction and accuracy in judgment, and provides a reliable tool for rapid determination of metadata validity.

[0113] In an optional implementation, it further includes:

[0114] Step d71: Obtain processing progress data in real time according to the preset collection cycle. The processing progress data includes the number of metadata integrity verification completed, the percentage of deduplication processing progress, the number of core feature information extraction nodes, the amount of hash operation completed, and the bitmap setting progress.

[0115] Specifically, processing progress data refers to the set of quantitative data representing the execution status of each stage in the construction of the effectiveness judgment filter; the preset collection period refers to the fixed time interval set by the storage system for periodically capturing progress information. Through the progress collection module, the key stages of filter construction are traversed according to the set period, and the quantitative execution data of each stage is collected in real time to form a structured processing progress dataset.

[0116] By dynamically capturing the real-time status of the filter construction process, comprehensive and accurate raw data support is provided for the subsequent generation of breakpoint information, avoiding the inability to accurately locate breakpoints due to missing progress information, realizing the visualization and traceability of the status of each processing stage, and providing the core data foundation for the implementation of the breakpoint mechanism.

[0117] Step d72: Generate scan breakpoint information based on processing progress data. The scan breakpoint information includes the previous generation snapshot identifier, the current processing stage marker, the index range of processed metadata, the generated partial fusion hash value, and the current state data of the bitmap.

[0118] Specifically, scan breakpoint information refers to structured data that records the critical processing status when filter construction is interrupted and is used for subsequent recovery processes. Through information integration algorithms, core recovery elements such as generation identifiers, current stages, data indexes, and intermediate results are extracted from the processing progress data and organized into scan breakpoint information according to a preset format.

[0119] By transforming scattered progress data into structured information with recovery guidance value, the processing nodes and intermediate results at the time of interruption are clearly identified, providing a clear basis for the continuation of the process after fault recovery. This solves the problem of no clear record of the processing status after interruption, and enables the interruption status constructed by the filter to be accurately reproduced, avoiding duplicate processing or omissions in the recovery process due to ambiguous status.

[0120] Step d73: ​​Associate the scan breakpoint information with the previous generation snapshot identifier and store it in the breakpoint storage partition of the database.

[0121] Specifically, the breakpoint storage partition refers to a dedicated storage area in the database used to store scan breakpoint information. A generational association mechanism binds the scan breakpoint information to the previous generation snapshot identifier, and then the associated breakpoint information is written to the dedicated partition through the database's persistence interface, ensuring the independence and security of data storage.

[0122] This system enables persistent storage and precise association of scan breakpoint information, preventing breakpoint information from being lost or confused with other generation data. It ensures that breakpoint information can be quickly retrieved through generation identifiers during fault recovery, solving the problems of easy loss and difficulty in locating breakpoint information. It ensures long-term stable storage of breakpoint information and clear association relationships, providing a reliable guarantee for subsequent rapid reading and recovery.

[0123] Step d74: If a controller failure or build process interruption is detected, the scan breakpoint information corresponding to the previous generation snapshot identifier is read from the database after the failure is recovered.

[0124] Specifically, the fault detection module monitors the controller's operating status and the execution status of the build process in real time. When an abnormal interruption is detected, after the system recovers, the previous generation snapshot is used as the search keyword, and the breakpoint storage partition is accessed through the database query interface to extract the corresponding scan breakpoint information.

[0125] Quickly locate and obtain the scan breakpoint information at the time of interruption, establish a bridge between fault recovery and interruption status, avoid the recovery process being without a basis, realize accurate retrieval and rapid reading of breakpoint information, lay the foundation for process continuation, and shorten recovery time.

[0126] Step d75: Based on the processing stage markers in the scan breakpoint information and the index range of processed metadata, skip metadata entries that have been processed.

[0127] Specifically, by parsing the current processing stage marker and the range of processed metadata indexes in the scan breakpoint information, data filtering is used to directly exclude metadata entries that have been processed within the index range in the recovered processing flow, and only unprocessed entries are retained to enter the subsequent process.

[0128] This effectively avoids redundant computation of already processed metadata, reduces unnecessary processing overhead, ensures that the recovery process only advances on the incomplete parts, improves the processing efficiency of the recovery process, reduces system resource consumption, and shortens the total time for filter construction.

[0129] Step d76: Load the generated fusion hash value and the current state data of the bitmap, and continue to execute the unfinished operations until the construction of the validity judgment filter corresponding to the previous generation snapshot identifier is completed.

[0130] Specifically, through the data loading interface, partial fused hash values ​​and current bitmap state data recorded in the scan breakpoint information are read and restored to the corresponding processing node in filter construction. Based on the restored intermediate data, unfinished hash operations, bit operations, or bitmap setting operations continue to be executed until the process is completely completed. This seamlessly connects the intermediate processing results before the interruption, ensuring that the filter construction process continues from the breakpoint, avoiding the invalidation of intermediate results due to interruption, achieving seamless continuation of the filter construction process, ensuring the integrity and accuracy of the filter, and improving the overall stability of the process.

[0131] This invention generates structured scan breakpoint information by periodically collecting and processing progress data, associates it with generation identifiers, and persistently stores it in a dedicated partition. After fault recovery, it accurately reads the breakpoint information, skips processed data, and loads intermediate results to continue the process. This effectively solves the problems of process interruption, data duplication, and loss of intermediate results caused by faults or interruptions during filter construction, ensuring the continuity, integrity, and efficiency of filter construction, and improving the stability and reliability of the entire processing solution.

[0132] This embodiment details the process of matching the metadata corresponding to the previous snapshot identifier in the database using a validity judgment filter, as described in the above embodiment. The specific implementation of this process includes the following steps:

[0133] Step m1: Obtain the core feature information of the metadata corresponding to the previous snapshot identifier in the database. The core feature information includes the location identifier and attribute features of the metadata.

[0134] Specifically, core feature information refers to the key data dimensions in the generational metadata that can uniquely represent its identity and attributes; location identifier refers to the unique code that marks the physical storage location of the metadata; attribute features refer to the inherent attribute information of the generational metadata, such as creation time and data type. Based on the previous generational snapshot identifier, the metadata of the corresponding generation is selected from the database, and the location identifier and attribute features are accurately extracted through feature extraction algorithms to form a standardized feature dataset.

[0135] By clarifying the core criteria for judging the validity of metadata, complex metadata is transformed into quantifiable feature information, providing a unified input for subsequent hash operations and matching operations. This ensures that the matching process focuses on the essential features of metadata, solves the problem of unclear matching criteria, achieves accurate extraction of core features, avoids matching deviations caused by improper feature selection, and lays a data foundation for matching accuracy.

[0136] Step m2 involves processing the location identifier and attribute characteristics of the metadata using a preset hash algorithm to obtain the third hash value and the fourth hash value.

[0137] Specifically, the preset hash algorithm refers to the same low-collision-rate hash calculation rule used when constructing the validity judgment filter. The extracted location identifier and attribute features are taken as inputs, and the hash algorithm performs a one-way mapping calculation, transforming the two types of high-dimensional features into fixed-length, evenly distributed third and fourth hash values.

[0138] This approach achieves dimensionality reduction and standardization of feature information, thereby reducing the computational complexity of subsequent matching operations. At the same time, by using hash logic consistent with the filter construction stage, it ensures the consistency of feature mapping, providing an algorithmic foundation for cross-step matching. This effectively solves the problems of low efficiency in high-dimensional feature processing and inconsistent cross-step mapping, forming a unified hash value format, improving the matching calculation speed, and ensuring the accuracy of feature mapping.

[0139] Step m3: Merge the third hash value and the fourth hash value according to the preset bitwise operation rules to obtain the hash value to be matched.

[0140] Specifically, the preset bit operation rules refer to the same feature fusion bit operation logic used when constructing the validity judgment filter. By performing operations on the binary bits of the third hash value and the fourth hash value, the hash information corresponding to the two types of features is merged into a single hash value to be matched, thus fully preserving the identity and attribute feature differentiation information of the metadata.

[0141] By integrating the hash information of the two types of features, a comprehensive feature mapping that can fully represent metadata is formed, avoiding the one-sidedness of matching caused by a single feature hash value, ensuring the integrity of the matching basis, realizing the deep integration of feature information, enabling the hash value to be matched to accurately reflect the essence of metadata, and improving the reliability of matching results.

[0142] Step m4: Determine the bitmap position corresponding to the filter based on the validity of the hash value to be matched.

[0143] Specifically, the bitmap is the core storage structure of the validity judgment filter, composed of a series of binary bits, with each bit corresponding to a hash value mapping position. Based on the preset bitmap length, a modulo operation is performed on the hash value to be matched to determine its unique mapping position in the filter bitmap, establishing a correspondence between hash values ​​and bitmap positions. This quickly establishes the association between the feature to be matched and the filter, replacing the inefficient traditional method of comparing each item one by one. It enables rapid location of the matching position, laying the structural foundation for efficient judgment, significantly shortening the matching position location time, and improving the overall speed of metadata validity judgment.

[0144] Step m5: Obtain the numerical status of the bitmap position. If the numerical status is a preset valid set state, it is determined that the metadata needs to be retained and moved to the new position.

[0145] Specifically, the preset valid setting state refers to the bit in the validity judgment filter that represents metadata as valid data. Figure 2 Number bit status (e.g., "1"). By accessing the target mapping position of the filter bitmap, its numerical status is read. If it matches the set status, the metadata is determined to be valid data, triggering the retention and transfer process.

[0146] Clearly define the criteria and processing path for valid metadata to ensure that valid data can be accurately identified and properly stored, avoid resource losses caused by the accidental destruction of valid data, solve the problem of ambiguous criteria for valid metadata, achieve accurate identification and targeted processing of valid data, and ensure the rational use of storage resources.

[0147] In step m6, if the numerical state is a preset invalid and unset state, then the metadata needs to be destroyed.

[0148] Specifically, the preset invalid unset state refers to the bit in the validity judgment filter that represents metadata as invalid data. Figure 2 Number base status (e.g., "0"). If the value status of the target mapping position matches this unset status, the metadata is determined to be invalid, triggering the destruction process to release storage resources.

[0149] Clearly define the criteria and processing path for invalid metadata to ensure that invalid data can be cleaned up in a timely manner, avoid the waste of space caused by invalid data occupying storage resources, solve the problems of vague criteria for invalid metadata judgment and untimely cleanup, achieve accurate identification and efficient cleanup of invalid data, and optimize storage resource utilization.

[0150] This invention extracts core features of metadata and generates hash values ​​to be matched using the same hash algorithm and bitwise operation rules as the filter construction. It quickly locates the position of the filter bitmap and determines the validity of metadata based on the numerical state, thus achieving accuracy and efficiency in metadata matching. This effectively solves the problems of low efficiency, ambiguous judgment criteria, and inconsistent logic across steps in traditional matching methods, ensuring the accuracy of metadata validity judgment results, and optimizing the recycling and utilization efficiency of storage resources.

[0151] In an optional implementation, the preset termination conditions include at least one of the following: the metadata validity judgment task of the storage system is completed, the metadata in the database has completed validity judgment and processing, a loop termination instruction issued by the user is received, or the storage system detects an abnormal fault and triggers an emergency termination of the loop process.

[0152] Specifically, the preset termination conditions refer to a set of pre-defined criteria used to trigger the termination of the metadata processing loop. These conditions are pre-built based on the storage system's operating scenarios and processing requirements, and include four core scenarios: task completion, data processing loop closure, external command intervention, and system fault emergency response. By monitoring the validity of metadata, the execution progress and results of the task are judged to confirm the achievement of core task objectives; by verifying the processing status of metadata within the processing scope, data processing is ensured to be complete without omissions; by receiving control commands issued by users, manual intervention in the process is supported; and by monitoring the storage system's operating status in real time, emergency termination is triggered in the event of abnormal failures to prevent the failure from escalating.

[0153] By clearly defining the triggering scenarios for the end of the metadata processing loop, a clear and executable termination judgment standard is provided for the loop process, avoiding incomplete data processing caused by the process falling into an infinite loop or premature termination; at the same time, it takes into account multiple scenarios such as normal business completion, manual intervention and fault emergency, ensuring the flexibility and security of the process, and ensuring that the system can orderly end the processing loop in different operating states.

[0154] It effectively solves the problems of ambiguous termination criteria and insufficient process controllability, so that the termination time of the metadata processing cycle has a clear basis, avoids invalid processing that occupies system resources, and prevents data corruption or resource waste caused by the continuous operation of the process in fault scenarios, thereby improving the feasibility and stability of the solution.

[0155] This invention clarifies the criteria for determining the end of a loop process by setting preset termination conditions covering four scenarios: task completion, data closure, user instructions, and abnormal faults. It balances the integrity, flexibility, and security of the process, effectively solving the problems of ambiguous termination criteria and insufficient process controllability, and ensuring that the metadata processing loop can end in an orderly and reliable manner in different scenarios.

[0156] In an optional implementation, the following steps are also included:

[0157] Step n1: Write the incremented current snapshot identifier into the database for persistent storage, forming a snapshot identifier history.

[0158] Specifically, snapshot identifier history refers to a structured data collection that stores incrementally increasing snapshot identifiers in chronological order. Through the database's persistent storage interface, the updated snapshot identifier is written to a dedicated metadata database, and an index is created by timestamp to form a historical record, ensuring the permanence and traceability of identifier storage.

[0159] This ensures that snapshot identifiers are not lost due to system restarts, malfunctions, or other factors, while forming a complete generation identifier traceability chain. This provides a basis for subsequent generation data verification and troubleshooting, solves the problems of easily lost snapshot identifiers and lack of historical traceability, ensures the continuity and traceability of generation division benchmarks, and provides stable historical data support for the entire generation management system.

[0160] Step n2: Obtain the garbage collection rate data of the storage system in real time, and calculate the average collection rate per unit time based on the garbage collection rate data.

[0161] Specifically, garbage collection rate data refers to the quantitative data on how quickly a storage system cleans up invalid metadata and releases storage resources per unit of time; average collection rate refers to the quantitative indicator obtained by averaging the garbage collection rate data over a period of time using a summarizing algorithm. The system monitoring module captures collection rate data in real time and uses a sliding window algorithm to calculate the average rate per unit of time, ensuring the real-time performance and stability of the data.

[0162] By quantifying the garbage collection efficiency of the storage system, data support is provided for subsequent adjustments to the timer interval, avoiding the mismatch between collection efficiency and system load caused by a fixed time interval. This enables dynamic monitoring and quantitative analysis of garbage collection efficiency, providing reliable data for adaptive adjustment of generation division cycles.

[0163] Step n3: Compare the average recovery rate with the preset rate threshold range.

[0164] Specifically, the preset rate threshold range refers to a reasonable rate range that balances recycling efficiency and system resource consumption, based on the storage system's hardware performance and business load characteristics. A numerical comparison algorithm compares the calculated average recycling rate with the preset threshold range, outputting the result determining the rate's position within that range.

[0165] By clarifying the rationality of the current waste recycling rate and establishing trigger criteria for rate adjustment, we can avoid blindly adjusting the timer interval, ensure that the adjustment operation has a clear basis, solve the problem of no judgment criteria for rate adjustment, and achieve accurate definition of the recycling rate status.

[0166] Step n4: If the average recycling rate is lower than the lower limit of the threshold range, shorten the preset time interval of the timer.

[0167] Specifically, the timer period is adjusted based on the rate comparison results, and the time interval is shortened by a preset adjustment range to accelerate the generation increment frequency of snapshot identifiers, thereby increasing the processing frequency of the previous generation of metadata.

[0168] By increasing the frequency of generation partitioning and the number of garbage collection executions, the problem of invalid data accumulation can be alleviated, the overall collection efficiency can be improved, the collection rate can be brought back to a reasonable range, the problem of excessive storage resource occupation caused by excessively low garbage collection rate can be solved, the collection rate can be dynamically improved, and the rational utilization of storage system resources can be ensured.

[0169] Step n5: If the average recycling rate is higher than the upper limit of the threshold range, then extend the preset time interval of the timer.

[0170] Specifically, based on the rate comparison results, the timer period adjustment logic is triggered to extend the time interval by a preset adjustment range, reduce the frequency of snapshot identifier generation increment, and reduce the frequency of garbage collection.

[0171] To avoid excessive system computing and storage resources being consumed due to excessive recycling frequency, balance the efficiency of garbage collection with the allocation of business processing resources, prevent excessive system load from affecting core business, solve the problem of system resource waste caused by excessive collection, achieve a dynamic balance between collection rate and system load, and ensure the overall stability of the storage system operation.

[0172] Step n6: When multiple controllers synchronize the current snapshot identifier, select one controller as the master controller and the rest as slave controllers.

[0173] Specifically, the master controller refers to the core controller in a multi-controller cluster that is responsible for synchronization coordination and command issuance; the slave controller refers to the subordinate controller that receives commands from the master controller and executes synchronization operations. Through a cluster election algorithm or preset screening rules, based on indicators such as controller load status and operational stability, the optimal controller is selected as the master controller, thus establishing a master-slave collaborative architecture.

[0174] The core coordinating entity for multi-controller synchronization is clearly defined to avoid synchronization chaos and command conflicts caused by the lack of unified command, improve the orderliness and efficiency of the synchronization process, solve the problem of the lack of a unified coordinating core when synchronizing multiple controllers, and establish an orderly master-slave collaboration mechanism to provide organizational support for subsequent synchronization command transmission and status feedback.

[0175] Step n7: The master controller sends the current snapshot identifier and synchronization command to the slave controller.

[0176] Specifically, based on a distributed cluster communication protocol, the master controller encapsulates the current snapshot identifier and synchronization execution instructions in a preset format and distributes them in batches to the slave controllers via an internal high-speed communication link, ensuring the integrity and real-time performance of data transmission. This unified transmission of core synchronization data and execution requirements ensures that the slave controllers clearly understand the synchronization target (current snapshot identifier) ​​and operational specifications, avoiding synchronization deviations caused by inconsistencies in information transmission. It achieves accurate and efficient distribution of synchronization information, resolving the problem of information asymmetry between multiple controllers and providing a unified and accurate input for slave controller synchronization operations.

[0177] Step n8: After receiving the snapshot identifier from the controller, verify the validity of the current snapshot identifier. If the verification is successful, update the current snapshot identifier in the local storage.

[0178] Specifically, the received snapshot identifiers are verified for format compliance and data integrity using the built-in validity verification algorithm of the controller. After confirming that the identifiers have not been tampered with or are error-free, the old identifiers are replaced through the local storage update interface to complete the local baseline synchronization. Invalid or erroneous snapshot identifiers are effectively filtered to avoid local generation baseline confusion caused by erroneous identifiers. This ensures that the generation baseline of the slave controller is completely consistent with that of the master controller, achieving accurate alignment of generation baselines of multiple controllers and avoiding metadata processing chaos caused by inconsistent baselines.

[0179] Step n9: The controller returns a synchronization confirmation signal to the main controller. After receiving the confirmation signal, the main controller determines that synchronization is complete.

[0180] Specifically, after the slave controller completes synchronization, it generates a confirmation signal for successful synchronization and sends it back to the master controller. The master controller, through a signal receiving and aggregation mechanism, confirms that all slave controllers have returned confirmation signals, and then determines that the entire synchronization process is complete, forming a closed-loop verification.

[0181] Ensure that the synchronization operation covers all participating controllers, avoid inconsistencies in the baseline caused by some controllers not being synchronized, ensure the uniformity of the generational baseline of the multi-controller cluster, realize closed-loop management of the synchronization process, ensure the consistency and integrity of the generational baseline of the multi-controller cluster, and provide reliable guarantees for subsequent distributed processing.

[0182] This invention provides a historical record by persistently storing snapshot identifiers to ensure the traceability of generation benchmarks. It dynamically adjusts the timer interval based on the garbage collection rate to balance collection efficiency and system load. It adopts a master-slave controller architecture, unified command issuance, validity verification, and closed-loop confirmation mechanism to ensure the synchronization consistency of multiple controllers. This effectively solves the problems of easy loss of snapshot identifiers, collection rate not adapting to system state, and chaotic synchronization of multiple controllers. It improves the stability of generation management, the adaptability of collection strategies, and the accuracy of distributed collaboration, providing more reliable support for the entire metadata processing solution.

[0183] In an optional implementation, the memory includes a first memory and a second memory; the first memory is used to store the memory space for the validity judgment filter being built; the second memory is used to store the memory space for the validity judgment filter that has been built and is used for metadata validity matching; the first memory and the second memory are allocated the same storage capacity; the first memory and the second memory alternately store the validity judgment filters that are being built and those that have been completed; the validity judgment filter that has been built is destroyed after the metadata validity matching is completed.

[0184] Specifically, the first memory is used to store the validity judgment filters during the construction process; the second memory is used to store the filters that have been built and are used for metadata validity matching. By allocating equal storage capacity to both memory blocks based on the filter storage requirements of the storage system, compatibility when switching between different memory blocks is ensured. Simultaneously, a memory partitioning mechanism clearly defines the functional boundaries of the two memory blocks, avoiding functional confusion. This solves the resource contention problem of filter construction and invocation under a single memory block, ensuring that the storage capacity of the two memory blocks is adapted to the filter requirements, and guaranteeing the smooth execution of subsequent switching processes.

[0185] Based on the incremental logic of snapshot identifiers, when a filter construction is initiated at a certain snapshot level, the system automatically allocates the construction task to the currently available first memory location. Intermediate data and final results are written to this memory location via a dedicated storage channel. During the construction process, the first memory location only responds to read and write requests and does not expose any external APIs. This ensures that the filter data being built is not affected by external calls, significantly improving the stability of the construction process and avoiding filter construction failures or accuracy deviations caused by data interference.

[0186] After the filter is built, the system updates the status of the first memory module from "under construction" to "effective" through a status switching mechanism, and exposes a call interface to the metadata matching module, allowing it to read the filter data in memory through a standardized protocol and perform validity matching operations. This connects the filter building and usage phases, clarifies the activation trigger logic after building, ensures that the matching module can obtain available filters in a timely manner, and guarantees the continuity of the metadata processing flow. It achieves seamless integration between filter building and usage, avoids interruptions in the matching process due to the lack of an activation mechanism, and improves the overall efficiency of metadata processing.

[0187] When a new-generation snapshot triggers filter construction, the system automatically allocates the construction task to the second memory (while the first memory is active) based on a dual-memory alternation rule. The second memory performs storage operations according to the same construction standards as the first memory, forming a parallel mode of "first memory active, second memory constructing." This achieves parallel execution of filter construction and usage, avoiding delays in filter invocation caused by the resource consumption of new-generation filter construction, and improving the system's concurrent processing capabilities. It also resolves the timing conflict between filter construction and usage, ensuring that the metadata matching process continues uninterrupted while new-generation filter construction remains undisturbed, significantly improving system processing efficiency.

[0188] After the second memory filter is built and switched to an active state, the system detects that the metadata matching task corresponding to the first memory filter has been completed. Through the lifecycle management mechanism, it triggers a clear instruction for the first memory filter, releasing the memory resources it occupies and restoring it to an idle state, awaiting the next round of build tasks. The system clearly defines the destruction trigger condition for the old version filter (the next-generation filter is built and active), ensuring the continued availability of the old version filter before the matching task is completed, while avoiding invalid filters occupying memory resources for extended periods. This solves the memory waste problem caused by the ambiguous timing of old version filter destruction, enabling the cyclical reuse of memory resources, while ensuring uninterrupted metadata matching and guaranteeing the continuity of the processing flow.

[0189] Driven by an incrementing snapshot identifier, the system continuously alternates between two memory blocks following a cyclical logic: first memory build - effective - second memory build - effective - first memory clear - standby - first memory rebuild. This forms a closed-loop process of build-effective-destroy-reuse, ensuring the long-term stable operation of filter construction and usage, achieving efficient utilization of memory resources, and supporting the storage system's continuous processing of metadata validity judgment requirements.

[0190] By establishing dual memory, independent storage carriers are provided for the two stages of filter construction and use, avoiding conflicts between construction and use caused by a single memory. Independent storage isolation is achieved during the filter construction process, preventing data writing, calculation and other operations during construction from interfering with the invocation of effective filters, and ensuring the integrity and stability of the construction data. Sustainable reuse of dual memory is achieved, ensuring the continuous supply of filters during multi-generational data processing and avoiding process stagnation caused by memory resource exhaustion.

[0191] This invention, through the configuration of a first memory and a second memory with independent functions and the same capacity, clearly defines that the two memory blocks respectively carry the functions of filter construction and activation. The dual memory is alternately reused by using the incremental snapshot identifier as the driving force, and the destruction is triggered by the completion of the construction of the next generation filter corresponding to the previous generation filter. The entire life cycle of filter construction, use, and destruction is clearly defined, which effectively solves the problems of resource conflicts in filter construction and use, ambiguous destruction timing of old version filters, and waste of memory resources. It ensures the continuity and stability of metadata validity matching, while improving the utilization rate and concurrent processing capability of system memory resources.

[0192] Figure 3 This is a schematic diagram of the metadata processing device based on data snapshots provided in this application. Figure 3 As shown, the metadata processing device 30 based on data snapshots includes:

[0193] The acquisition module 301 is used to acquire the metadata validity judgment requirements of the storage system and the request data of the host. The request data is the metadata to be processed corresponding to the write input / output request issued by the host.

[0194] The identification module 302 is used to start a timer based on the validity judgment requirement, generate an initial snapshot identifier, and increment it sequentially at a preset time interval to obtain the current snapshot identifier.

[0195] The embedding module 303 is used to embed the current snapshot identifier into the key of the request data, obtain the target metadata with time generation mark, and store it in the database for persistent processing.

[0196] Synchronization module 304 is used to synchronize the current snapshot identifiers among multiple controllers of the storage system.

[0197] The flush module 305 is used to perform a forced flush operation on the non-flushable target metadata in the cache that corresponds to the previous generation snapshot identifier, so as to obtain the generational metadata.

[0198] Module 306 is used to construct a validity judgment filter corresponding to the previous generation snapshot identifier based on the generational metadata.

[0199] The processing module 307 is used to match the metadata corresponding to the previous generation snapshot identifier in the database through a validity judgment filter, obtain the metadata processing result, and return it to the timer for the next generation processing until the preset termination condition is met. The metadata processing result includes retaining the previous generation metadata and storing it in a new location or destroying it.

[0200] In one possible design, the identification module 302 is specifically used for:

[0201] Set the initial snapshot identifier to a preset baseline value and store it in the database.

[0202] The trigger status of the timer is monitored, and when the timer reaches the preset time interval, the current snapshot identifier is incremented sequentially.

[0203] Update the incremented snapshot identifier to the current snapshot identifier.

[0204] In one possible design, the embedded module 303 is specifically used for:

[0205] Parse the key structure of the request data to obtain the identifier embedding field used to carry time generation information.

[0206] The current snapshot identifier is converted into a standard identifier according to the storage system's preset encoding specifications.

[0207] Write the standard identifier into the identifier embedding field to obtain the target metadata with time generation tags.

[0208] In one possible design, the brush module 305 is specifically used for:

[0209] Retrieve the target metadata that has not been refreshed from the cache, corresponding to the previous generation snapshot identifier.

[0210] The unflushd target metadata is written to the database through forced flushing, and the unflushd target metadata is written to the corresponding storage partition in the database in blocks synchronously through a parallel write algorithm.

[0211] Obtain the database feedback on the write status of the target metadata that has not been flushed, and confirm that the target metadata has been stored successfully.

[0212] The target metadata corresponding to the previous generation snapshot identifier stored in the storage partition is summarized to obtain generational metadata.

[0213] In one possible design, module 306 is specifically used for:

[0214] Perform integrity checks on the generational metadata and delete invalid or corrupt metadata records.

[0215] Perform deduplication on the validated metadata and retain only the unique and valid metadata entry.

[0216] Extract the core feature information of the metadata entries, which includes location identifiers and attribute features.

[0217] The location identifier and attribute features are processed by a preset hash algorithm to obtain the first hash value and the second hash value.

[0218] The first hash value and the second hash value are merged according to a preset bitwise operation rule to obtain a merged hash value.

[0219] The merged hash value is mapped to a bitmap of a preset length, and the values ​​at the mapped positions are set to generate a validity filter.

[0220] In one possible design, building module 306 is also used for:

[0221] The processing progress data is acquired in real time according to the preset collection cycle. The processing progress data includes the number of metadata integrity verification completed, the percentage of deduplication progress, the number of core feature information extraction nodes, the amount of hash operation completed, and the bitmap setting progress.

[0222] Scan breakpoint information is generated based on processing progress data. The scan breakpoint information includes the previous generation snapshot identifier, the current processing stage marker, the index range of processed metadata, the generated partial fusion hash value, and the current state data of the bitmap.

[0223] The scan breakpoint information is associated with the previous generation snapshot identifier and stored in the breakpoint storage partition of the database.

[0224] If a controller failure or build process interruption is detected, the scan breakpoint information corresponding to the previous generation snapshot identifier is read from the database after the failure is recovered.

[0225] Based on the processing stage markers and the index range of processed metadata in the scan breakpoint information, skip metadata entries that have been processed.

[0226] Load the generated fusion hash value and the current state data of the bitmap, and continue to execute the unfinished operations until the construction of the validity judgment filter corresponding to the previous generation snapshot identifier is completed.

[0227] In one possible design, processing module 307 is specifically used for:

[0228] Obtain the core feature information of the metadata corresponding to the previous snapshot identifier in the database. The core feature information includes the location identifier and attribute features of the metadata.

[0229] The location identifier and attribute characteristics of the metadata are processed by a preset hash algorithm to obtain the third hash value and the fourth hash value.

[0230] The third hash value and the fourth hash value are merged according to the preset bitwise operation rules to obtain the hash value to be matched.

[0231] The position of the bitmap corresponding to the filter is determined based on the validity of the hash value to be matched.

[0232] Get the numerical status of the bitmap position. If the numerical status is a preset valid set state, then determine that the metadata needs to be retained and moved to the new location.

[0233] If the numerical status is a preset invalid and unset state, then the metadata needs to be destroyed.

[0234] In one possible design, the preset termination conditions include at least one of the following: the metadata validity judgment task of the storage system is completed, the metadata in the database has completed validity judgment and processing, a loop termination instruction issued by the user is received, or the storage system detects an abnormal fault and triggers an emergency termination of the loop process.

[0235] In one possible design, the synchronization module 304 is also used for:

[0236] The incremented current snapshot identifier is written to the database for persistent storage, forming a snapshot identifier history.

[0237] The system acquires garbage collection rate data from the storage system in real time and calculates the average collection rate per unit time based on the garbage collection rate data.

[0238] The average recovery rate is compared with the preset rate threshold range.

[0239] If the average recovery rate is lower than the lower limit of the threshold range, the preset time interval of the timer will be shortened.

[0240] If the average recycling rate is higher than the upper limit of the threshold range, the preset time interval of the timer will be extended.

[0241] When multiple controllers synchronize the current snapshot identifier, one controller is selected as the master controller and the rest are slave controllers.

[0242] The master controller sends the current snapshot identifier and synchronization command to the slave controller.

[0243] Upon receiving the snapshot identifier from the controller, the validity of the current snapshot identifier is verified. If the verification is successful, the current snapshot identifier in the local storage is updated.

[0244] The controller sends a synchronization confirmation signal to the main controller, and the main controller determines that synchronization is complete after receiving the confirmation signal.

[0245] For a description of the features in the embodiment of the metadata processing device based on data snapshots, please refer to the relevant description of the embodiment of the metadata processing method based on data snapshots, which will not be repeated here.

[0246] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the electronic device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus.

[0247] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to execute the above-described metadata processing method embodiment based on data snapshot.

[0248] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0249] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0250] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0251] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0252] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the metadata processing method based on data snapshots at runtime.

[0253] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0254] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the metadata processing method based on data snapshots.

[0255] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the metadata processing method based on data snapshots.

[0256] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of this application.

[0257] The foregoing has provided a detailed description of a metadata processing method and electronic device based on data snapshots provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A metadata processing method based on data snapshots, characterized in that, include: Obtain the metadata validity judgment requirements of the storage system and the request data of the host, wherein the request data is the metadata to be processed corresponding to the write input / output request issued by the host; Based on the validity judgment requirement, a timer is started to generate an initial snapshot identifier and increments it at preset time intervals to obtain the current snapshot identifier. The current snapshot identifier is embedded in the key of the request data to obtain target metadata with time generation mark and stored in the database for persistent processing; Synchronize the current snapshot identifiers among multiple controllers of the storage system; Force flush the cached target metadata that has not been flushed and corresponds to the previous generation snapshot identifier to obtain generational metadata; Based on the generational metadata, metadata entries are obtained. The fused hash value corresponding to the core feature information of the metadata entries is mapped to a bitmap of a preset length. The values ​​at the mapped positions are set to generate a validity judgment filter corresponding to the previous generation snapshot identifier. The validity filter matches the metadata corresponding to the previous generation snapshot identifier in the database to obtain the metadata processing result, and returns it to the timer for the next generation processing until the preset termination condition is met. The metadata processing result includes retaining the previous generation metadata and storing it in a new location or destroying it. The validity filter matches the metadata corresponding to the previous snapshot identifier in the database, including: Obtain the core feature information of the metadata corresponding to the previous generation snapshot identifier in the database. The core feature information includes the location identifier and attribute features of the metadata. The location identifier and attribute features of the metadata are processed by a preset hash algorithm to obtain a third hash value and a fourth hash value; The third hash value and the fourth hash value are merged according to a preset bitwise operation rule to obtain the hash value to be matched; The bitmap position corresponding to the validity determination filter is located based on the hash value to be matched. Obtain the numerical state of the bitmap position. If the numerical state is a preset valid set state, then determine that the metadata needs to be retained and moved to a new position. If the numerical state is a preset invalid and unset state, then the metadata needs to be destroyed.

2. The metadata processing method based on data snapshots according to claim 1, characterized in that, Based on the validity judgment requirement, a timer is started to generate an initial snapshot identifier and increment it sequentially at preset time intervals to obtain the current snapshot identifier, including: Set the initial snapshot identifier to a preset baseline value and store it in the database; The triggering status of the timer is monitored, and when the timer reaches a preset time interval, the current snapshot identifier is incremented sequentially. Update the incremented snapshot identifier to the current snapshot identifier.

3. The metadata processing method based on data snapshots according to claim 1, characterized in that, Embedding the current snapshot identifier into the key of the request data includes: The key structure of the request data is parsed to obtain an identifier embedding field used to carry time generation information; The current snapshot identifier is format-converted according to the preset encoding standard of the storage system to obtain a standard identifier; Write the standard identifier into the identifier embedding field to obtain the target metadata with time generation mark.

4. The metadata processing method based on data snapshots according to claim 1, characterized in that, Perform a forced flush operation on the target metadata in the cache that corresponds to the previous generation snapshot identifier and has not been flushed, including: Retrieve the target metadata that has not been refreshed from the cache, corresponding to the previous generation snapshot identifier; The un-flushd target metadata is written to the database through forced flushing, and the un-flushd target metadata is written to the corresponding storage partition in the database in blocks synchronously through a parallel write algorithm; Obtain the write status of the target metadata that has not been refreshed from the database, and confirm that the target metadata that has not been refreshed has been stored. The target metadata corresponding to the previous generation snapshot identifier stored in the storage partition is summarized to obtain generational metadata.

5. The metadata processing method based on data snapshots according to claim 1, characterized in that, Based on the generational metadata, metadata entries are obtained. The fused hash values ​​corresponding to the core feature information of the metadata entries are mapped to a bitmap of a preset length. Bit setting processing is performed on the values ​​at the mapped positions to generate a validity judgment filter corresponding to the previous generation snapshot identifier, including: Perform integrity verification on the generational metadata and delete invalid or corrupt metadata records; Perform deduplication on the verified generational metadata, retaining only the unique and valid metadata entry; Extract the core feature information of the metadata entry, which includes location identifier and attribute features; The location identifier and the attribute feature are processed by a preset hash algorithm to obtain a first hash value and a second hash value; The first hash value and the second hash value are merged according to a preset bitwise operation rule to obtain a merged hash value; The fused hash value is mapped to a bitmap of a preset length, and the values ​​at the mapped positions are set to generate a validity judgment filter.

6. The metadata processing method based on data snapshots according to claim 5, characterized in that, Also includes: The processing progress data is acquired in real time according to a preset collection cycle. The processing progress data includes the number of metadata integrity verification completed, the percentage of deduplication processing progress, the number of core feature information extraction nodes, the amount of hash operation completed, and the bitmap setting progress. Based on the processing progress data, scan breakpoint information is generated. The scan breakpoint information includes the previous generation snapshot identifier, the current processing stage marker, the index range of processed metadata, the generated partial fusion hash value, and the current state data of the bitmap. The scan breakpoint information is associated with the previous generation snapshot identifier and stored in the breakpoint storage partition of the database; If a controller failure or build process interruption is detected, the scan breakpoint information corresponding to the previous generation snapshot identifier is read from the database after the failure is recovered. Based on the processing stage markers and the index range of processed metadata in the scan breakpoint information, skip the metadata entries that have been processed. Load the generated fusion hash value and bitmap current state data, and continue to execute the unfinished operations until the construction of the validity judgment filter corresponding to the previous generation snapshot identifier is completed.

7. The metadata processing method based on data snapshots according to claim 1, characterized in that, The preset termination conditions include at least one of the following: the metadata validity judgment task of the storage system is completed, the metadata in the database has been validated and processed, a loop termination instruction issued by the user is received, or the storage system detects an abnormal fault and triggers an emergency termination of the loop process.

8. The metadata processing method based on data snapshots according to claim 1, characterized in that, Also includes: The incremented current snapshot identifier is written to the database for persistent storage, forming a snapshot identifier history record; The garbage collection rate data of the storage system is acquired in real time, and the average collection rate per unit time is calculated based on the garbage collection rate data. The average recovery rate is compared with a preset rate threshold range; If the average recovery rate is lower than the lower limit of the threshold range, then the preset time interval of the timer is shortened; If the average recovery rate is higher than the upper limit of the threshold range, then the preset time interval of the timer is extended; When multiple controllers synchronize the current snapshot identifier, one controller is selected as the master controller and the remaining controllers are slave controllers. The master controller sends the current snapshot identifier and synchronization command to the slave controller; Upon receiving the data from the controller, the validity of the current snapshot identifier is verified. If the verification is successful, the current snapshot identifier stored locally is updated. The slave controller returns a synchronization confirmation signal to the master controller, and the master controller determines that synchronization is complete after receiving the confirmation signal.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the metadata processing method based on data snapshots as described in any one of claims 1 to 8 when executing the computer program.

Images