Storage unit verification method and electronic device

Abstract

The application discloses a storage unit verification method and electronic equipment, and relates to the technical field of artificial intelligence, which comprises the following steps: reading the head information of a storage unit to be verified, analyzing and verifying the metadata field in the head information to obtain a head verification result; after the head verification, determining the decision score of the storage unit to be verified based on at least one strategy factor; in response to the decision score being greater than or equal to a preset decision threshold, reading the tail metadata of the storage unit to be verified, performing consistency check in combination with the head information, performing validity check on each data block in the storage unit to be verified that passes the check, identifying valid data blocks and marking the valid data blocks, and updating the pre-verification cache, so as to solve the technical problems of insufficient reliability guarantee, large performance overhead and inability to cope with the scene in which part of the data is valid caused by the single FU check method, and achieve the technical effects of improving the data verification strength while taking into account the system performance and storage efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a storage cell verification method and electronic device. Background Technology

[0002] Related technologies typically employ a validity verification method based on a single check of the FU (Flush Unit) header. The basic process involves a background task reading the header information and calculating a Cyclic Redundancy Check (CRC) code when it iterates through an FU, then comparing it with the CRC code stored in the header. If the comparison is successful, the process continues to process the physical block addresses within the FU; otherwise, it skips or marks the invalid physical block address. While this method is simple and direct, it has the following significant drawbacks: First, the verification strength is insufficient and reliability is low. Relying solely on a single CRC code cannot detect errors in the data within the FU or logical errors caused by partial damage to the header itself. Second, the verification strategy is simplistic and inefficient. The verification process does not consider factors such as the frequency of FU usage or system load, applying the same verification strength to data with varying usage frequency, which can lead to wasted performance and negatively impact system performance under high load. Third, it cannot handle scenarios where some data is valid. If an FU contains partially valid data, current technology cannot distinguish between them. If invalid data is detected within the FU, the entire FU is discarded, potentially wasting valid data space.

[0003] Therefore, in view of the shortcomings of related technical solutions, the present invention provides a storage cell verification method. Summary of the Invention

[0004] This application provides a storage cell verification method and electronic device to at least solve the problems of insufficient reliability guarantee, high performance overhead and inability to handle scenarios where some data is valid due to the single FU verification method in related technologies.

[0005] This application provides a storage unit verification method, the method comprising: reading the header information of the storage unit to be verified, parsing the metadata fields in the header information, verifying the metadata fields according to preset verification rules, and obtaining a header verification result; in response to the header verification result being a pass verification, determining a decision score for the storage unit to be verified based on at least one policy factor among the current system load, historical failure count, and business fault tolerance requirements; in response to the decision score being less than a preset decision threshold, using the header verification result as the verification result for the storage unit to be verified, and updating the pre-verification cache; in response to the decision score being greater than or equal to the preset decision threshold, reading the tail metadata of the storage unit to be verified, performing consistency verification in conjunction with the header information, and performing validity verification on each data block in the storage unit to be verified that has passed the verification, identifying and marking the valid data blocks of the storage unit to be verified, and updating the pre-verification cache.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described memory cell verification methods when executing the computer program.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described storage unit verification methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described memory cell verification methods.

[0009] This application involves reading the header information of the storage unit to be verified, parsing the metadata fields in the header information, and verifying the metadata fields according to preset verification rules to obtain a header verification result. If the header verification result is successful, a decision score for the storage unit to be verified is determined based on at least one policy factor among the current system load, historical failure count, and business fault tolerance requirements. If the decision score is less than a preset decision threshold, the header verification result is used as the verification result for the storage unit to be verified, and the pre-verification cache is updated. If the decision score is greater than or equal to the preset decision threshold, the tail metadata of the storage unit to be verified is read, and consistency verification is performed in conjunction with the header information. For storage units that pass the verification... Each data block in the storage unit to be verified undergoes validity verification. Valid data blocks in the storage unit to be verified are identified and marked, and the pre-verification cache is updated. Therefore, by introducing a pre-verification cache and a dynamic decision-making mechanism, system performance is optimized while ensuring data reliability. A three-level progressive verification approach, consisting of header, tail, and data block verification, effectively identifies underlying data errors that cannot be detected by a single header verification, significantly improving data integrity. The verification depth is dynamically determined by a policy factor, achieving a flexible balance between lightweight verification under high load and deep verification under low load. Furthermore, by marking some valid data blocks, the system can finely manage storage space, reclaiming only invalid areas, thereby improving storage utilization and system fault tolerance. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic flowchart illustrating a storage unit verification method provided in an embodiment of this application; Figure 2A schematic diagram of the FU data architecture for a storage unit verification method provided in this application embodiment; Figure 3 This is a schematic diagram of a multi-level verification process for a storage unit verification method provided in an embodiment of this application; Figure 4 A schematic diagram of the dynamic multi-level verification processing timing of a storage unit verification method provided in this application embodiment; Figure 5 A structural block diagram of a storage cell verification device provided in an embodiment of this application; Figure 6 This is an internal structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] It should be noted that the terms "S1," "S2," etc., are used only for descriptive purposes and do not specifically refer to the order or sequence, nor are they intended to limit this application. They are merely for the convenience of describing the method of this application and should not be construed as indicating the sequential order of the steps. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The embodiments of this application provide a storage cell verification method, and the method is described in detail in conjunction with the execution flow of the storage cell verification method.

[0017] S101: Read the header information of the storage unit to be verified, parse the metadata fields in the header information, verify the metadata fields according to the preset verification rules, and obtain the header verification result.

[0018] Here, the storage unit to be verified can be a flush unit (FU), a key data structure. An FU is the smallest unit in the storage system for data flushing and writing operations. It contains a header and several physical block addresses (PBAs). The header stores metadata for the FU, such as version number, volume ID, and cyclic redundancy check (CRC) code. Its core purpose is to aggregate random, small-granularity I / O into sequential, large-granularity writes, thereby improving flushing efficiency.

[0019] The storage unit to be verified can also be a Chunk, an Extent, an Object, etc., which is applicable to all scenarios that require storage unit validity verification.

[0020] The header information refers to a structured metadata area located at the beginning of the FU, which is used to store key information required to describe, manage, and verify the FU.

[0021] The header information may include version number, magic number, CRC (Cyclic Redundancy Check), etc.

[0022] S102: In response to the header verification result being verified as passed, determine the decision score of the storage unit to be verified based on at least one of the following policy factors: current system load, number of historical failures, and business fault tolerance requirements.

[0023] Here, the current system load refers to the I / O operation pressure or bandwidth usage that the storage system experiences per unit of time, and is a key indicator reflecting the real-time busyness of the system.

[0024] Here, the number of historical failures includes the cumulative number of times FU has experienced verification anomalies or data corruption in the past.

[0025] Specifically, in response to a header verification result indicating that the verification failed, the storage unit to be verified is directly marked as invalid.

[0026] S103: In response to a decision score that is less than a preset decision threshold, the header verification result is used as the verification result of the storage unit to be verified, and the pre-verification cache is updated.

[0027] Specifically, if the decision score is below the threshold, it means that the verification of the storage unit to be verified needs to prioritize performance, that is, the decision result is to adopt a method that relies solely on header CRC check.

[0028] S104: In response to a decision score greater than or equal to a preset decision threshold, read the tail metadata of the storage unit to be verified, perform consistency verification in combination with the header information, and perform validity verification on each data block in the storage unit to be verified that has passed the verification, identify and mark the valid data blocks of the storage unit to be verified, and update the pre-verification cache.

[0029] Here, tail reverse verification refers to a two-way verification mechanism that stores the reverse checksum (such as SHA-256 hash value) of the header information in the last data block of the write unit, and recalculates the header hash value during verification and compares it with the tail stored value.

[0030] Here, data block verification refers to the independent verification of the content of each independent data block (grain) within the FU. The validity of each block is determined by comparing the stored checksum with the real-time calculated checksum.

[0031] Specifically, if the decision score is higher than the threshold, it means that the storage unit to be verified undergoes multi-level deep verification to ensure reliability, that is, the decision result is to perform multi-level deep verification.

[0032] In one embodiment, the decision factor further includes storage unit replica configuration information; in response to the storage unit to be verified being configured in multi-replica mode, a multi-replica collaborative verification process is executed to identify and mark valid data blocks; in response to the storage unit to be verified being configured in single-replica mode, a single-replica deep verification process is executed to identify and mark valid data blocks. Specifically, in response to the storage unit to be verified being configured in multi-replica mode, data block verification is performed on the storage unit to be verified, valid data blocks are identified and marked, including: Read data from the same storage unit to be verified from at least two copies in parallel; For at least two replicas, perform data block verification operations to obtain a valid data bitmap for each replica; Responding to at least two copies of valid data bits Figure 1 The data block verification result is confirmed as passed. In response to inconsistencies in the valid data bitmaps of at least two copies, a new copy is introduced, and the data block verification result is determined by majority vote.

[0033] Here, the multi-replica mode means that the same logical data is synchronously saved as two or more identical copies in physical storage.

[0034] Here, parallel reading involves simultaneously initiating multiple independent read requests to the storage system to obtain different physical copies of the same logical data.

[0035] The majority rule principle states that when there is a disagreement among the validity verification results of multiple copies, the verification results of an additional copy are introduced, and the consensus state that accounts for a majority (more than 50%) is adopted as the final decision.

[0036] This significantly enhances the credibility and authority of the verification results, and establishes an intelligent data self-healing trigger mechanism.

[0037] It should be noted that this application optimizes system performance while ensuring data reliability by introducing pre-verification caching and dynamic decision-making mechanisms; it adopts a three-level progressive verification of header, tail, and data blocks to effectively identify underlying data errors that cannot be detected by a single header verification, significantly improving data integrity; it dynamically determines the verification depth through policy factors, achieving a flexible balance between lightweight verification under high load and deep verification under low load; in addition, by marking some valid data blocks, the system can finely manage storage space, only reclaiming invalid areas, thereby improving storage utilization and system fault tolerance.

[0038] In some specific implementations, before querying the pre-verification cache and determining the cache verification result to read the header information of the storage unit to be verified, the method further includes: Obtain the identifier of the storage unit to be verified, query the pre-verification cache, and determine whether the storage unit to be verified exists in the pre-verification cache; In response to the presence of a storage unit to be verified in the pre-verification cache, the timestamp corresponding to the storage unit to be verified is extracted, compared with the current system time, and the lifespan is determined. In response to the storage unit to be verified having a lifespan less than or equal to a preset lifespan, the valid data bitmap of the storage unit to be verified is obtained, the cache verification result is determined to be valid, and the information in the cache verification result is used as the verification result of the storage unit to be verified.

[0039] Here, the pre-validation cache is global and implemented based on a hash table.

[0040] The pre-validation cache may include the identifier of the FU, the latest validation result of the FU, a bitmap recording valid data blocks, and a timestamp.

[0041] The pre-verification cache may also include verification strength level, cache hit count, and space usage.

[0042] Here, the identifier of the storage unit to be verified is the unique identifier of the write unit (FU) in the storage system. It is usually generated by combining logical volume ID, physical address information, serial number, etc., to ensure that it is uniquely identifiable globally.

[0043] The storage structure of the pre-verification cache is a key-value pair, where the key is the identifier of the storage unit to be verified, and the value is the verification result, valid data bitmap, timestamp, and cache lifespan.

[0044] Here, the preset lifespan is the maximum valid duration of a pre-configured cache entry; exceeding this time is considered cache expiration.

[0045] Here, the valid data bitmap is a binary bitmap, where each bit corresponds to a data block (grain) in FU, with 1 indicating valid and 0 indicating invalid.

[0046] The preset lifetime can be dynamically adjusted based on FU characteristics and system policies. For example, the preset lifetime for hot data FUs is shorter (e.g., 5 minutes) to ensure timely updates; the preset lifetime for cold data FUs is longer (e.g., 1 hour) to reduce verification frequency; and the preset lifetime for critical volume FUs is medium to balance reliability and performance.

[0047] Among them, cache invalidity includes miss and expired; a miss can be that there is no identifier of the storage unit to be verified in the cache verification result, and expired can be that the storage time corresponding to the storage unit to be verified in the cache verification result exceeds the valid time.

[0048] Specifically, if the cache verification result satisfies a cache hit and the cache has not expired, the information in the cache verification result is directly used as the verification result for the storage unit to be verified. If the cache verification result is that the cache is invalid, subsequent execution reads the header information of the storage unit to be verified.

[0049] This improves cache hit rate and accuracy, avoiding redundant I / O and computation.

[0050] In some specific implementations, the header information of the storage unit to be verified is read, the metadata fields in the header information are parsed, and the metadata fields are verified according to preset verification rules. Header verification of the storage unit to be verified is performed to obtain the header verification result, including: Read the header information of the storage unit to be verified, and parse it to obtain the magic number field, version number field, and cyclic redundancy check code field; Compare the magic number field with the valid magic number field value to obtain the first verification result; In response to the first verification result being a match, the version number field is compared with the range of valid version number fields to obtain the second verification result; In response to the second verification result being a match, the third calculated field is determined based on the header information. The third calculated field is then compared with the cyclic redundancy check code field to obtain the third verification result. If the third verification result is a match, the header verification result is determined to be a pass.

[0051] Here, the magic number field is a predefined fixed value used to quickly identify the starting position and basic format of the FU. Valid magic number field values ​​are legal magic numbers.

[0052] Here, the version number field represents the version number of the FU data structure, used for compatibility management. The valid version number field is a range of valid version numbers.

[0053] Here, the Cyclic Redundancy Check (CRC) field is a CRC check value calculated from part or all of the header content, used to check the integrity of the header information.

[0054] Specifically, the starting physical address of the FU is located, the first data block of the FU is read from the storage medium, the fields of the FU are parsed according to the predefined header structure, the parsed magic number field value is obtained, and it is compared bit by bit with the system's predefined valid magic number value to determine whether they match completely. The parsed version number field value is obtained and compared with the valid version number range supported by the system to determine whether it is within the valid range. The CRC calculation range is determined, and the checksum is calculated using the same CRC algorithm as when it was stored (such as CRC32, CRC64). The calculated value is compared with the CRC field value stored in the header. Only when the magic number, version, and CRC verifications all pass, is the header verification considered successful. If any step of the verification fails, the verification process is immediately terminated, and the FU is marked as invalid. After the verification is successful, the FU enters the subsequent dynamic decision-making stage.

[0055] In this way, structural damage or version mismatch issues of storage units can be detected at the earliest stage with minimal overhead, thereby immediately terminating the subsequent processing of invalid units and avoiding unnecessary waste of resources.

[0056] In some specific implementations, a decision score for the storage unit to be verified is determined based on at least one policy factor among the current system load, historical failure count, and business fault tolerance requirements, including: Collect the current number of input / output operations per second, normalize the operations, and obtain the load factor; Based on the number of historical failures of the storage unit to be verified, the number of historical failures is mapped to a normalized interval to determine the historical failure factor. Based on the data fault tolerance requirements of the storage unit to be verified, graded values ​​are assigned to determine the fault tolerance factor. The decision score is obtained by weighting the load factor, historical failure factor, and fault tolerance factor.

[0057] Here, decision factors refer to a set of multi-dimensional evaluation indicators that influence the decision on verification strength, used to quantify the current operating status of the system, data characteristics, and business importance.

[0058] Here, normalization is a mathematical process that converts the original value into a standardized value in the range [0,1].

[0059] Here, the decay function is a mathematical function that maps the time difference to the interval (0,1].

[0060] The preset logical volume priority configuration table is a predefined or administrator-configured static table that records the priority weight of each logical volume.

[0061] Specifically, the load factor is a value between 0 and 1 (0 represents idle and 1 represents full load) obtained by dividing the current number of I / O operations per second (IOPS) or bandwidth utilization of the system by a preset maximum value. This value is used to indicate the current load severity of the system.

[0062] Specifically, the historical failure factor is a statistical count of the cumulative number of times the FU has experienced verification anomalies or data corruption in the past, mapped to the range of 0 to 1 according to the failure frequency; the higher the failure frequency, the larger the value, which means that deep verification is more necessary.

[0063] Specifically, the fault tolerance factor is assigned a graded value based on the data fault tolerance requirements of the business carried by the FU. The fault tolerance level is 1.0 for core financial / critical business, 0.5 for ordinary online business, and 0.2 for archived cold data. The higher the fault tolerance requirement, the more priority is given to deep verification.

[0064] Specifically, the decision score = α × load factor + β × historical failure factor + γ × fault tolerance factor, where α, β, and γ are configurable weight coefficients. For example, α is 0.3, β is 0.4, and γ is 0.3.

[0065] In one embodiment, a multi-layer decision model can be used to calculate decision scores. This model includes: a first layer: basic factor calculation (load, failure frequency, fault tolerance requirements); a second layer: extended factor integration (media health, error history, etc.); a third layer: context-aware adjustment (business time period, energy consumption strategy, etc.); and a fourth layer: comprehensive decision output. Each layer can use different weight configurations, and feedback mechanisms can be set between layers to optimize parameters based on historical decision performance.

[0066] This allows the verification intensity to change adaptively, making the decision-making more objective and quantifiable.

[0067] In some specific implementations, the tail metadata of the storage unit to be verified is read, and consistency verification is performed in combination with the header information. The validity of each data block in the storage unit to be verified that passes the verification is then checked, and valid data blocks in the storage unit to be verified are identified and marked. This includes: Read the tail data block of the storage unit to be verified and parse it to obtain the reverse checksum of the storage header information; Read the current header information of the storage unit to be verified, calculate the hash value using a preset hash algorithm, and obtain the header checksum. The header checksum and the reverse checksum of the header information are compared to obtain the comparison result; In response to a consistent comparison result, the validity of each data block in the storage unit to be verified is checked, and the valid data blocks of the storage unit to be verified are identified and marked.

[0068] Here, the tail data block is the last physical data block of the write unit (FU), which is specifically used to store metadata such as reverse check codes.

[0069] Here, the header information reverse checksum is a cryptographic hash value stored in the tail data block, which is calculated and generated from the header information when the FU is created.

[0070] The hash algorithm can be SHA-256, SHA-384, SHA-512, etc.

[0071] In this way, the information at the beginning and end of the code is cross-checked, which improves the ability to detect malicious or accidental data corruption and significantly enhances the robustness and depth of the verification mechanism.

[0072] In some specific implementations, the validity of each data block in the storage unit to be verified that has passed the verification is checked, and the valid data blocks of the storage unit to be verified are identified and marked, including: From the header information of the storage unit to be verified, obtain the pre-stored checksum list of data blocks, where the pre-calculated checksum of each data block is stored in the data block checksum list; Initialize the valid data bitmap, wherein the number of bits in the valid data bitmap is the same as the total number of data blocks contained in the storage unit to be verified; Traverse each data block in the storage unit to be verified, perform verification operations on each data block, and obtain the data block verification results; In response to a data block validation result of passing the validation, the corresponding position in the valid data bitmap is marked as valid; After the verification operation of each data block is completed, check the status of the valid data bitmap.

[0073] Here, the data block checksum list is an ordered data structure stored in the header of the write unit (FU), containing pre-calculated checksum values ​​for each data block within the FU. The data block checksum list is arranged in data block order, with each entry corresponding to a pre-stored checksum for one data block, used for subsequent verification of the integrity of the data block content.

[0074] The data block checksum list typically follows the header metadata and occupies one or more data blocks.

[0075] Specifically, when the checksum comparison of a data block is successful, the corresponding position is set to '1' (valid); otherwise, it is set to '0' (invalid). After all data blocks have been checked, if the bitmap is all '1', FU is marked as "fully valid"; if the bitmap is all '0', it is marked as "invalid"; if the bitmap contains both '1' and '0', it is marked as "partially valid".

[0076] In this way, block-level validity identification is achieved.

[0077] In some specific implementations, a verification operation is performed on each data block to obtain the data block verification result, including: Read the contents of the current data block and calculate the real-time checksum corresponding to the current data block using a preset checksum algorithm; Compare the real-time checksum with the pre-stored checksum to determine the checksum comparison result; If the checksum and comparison results are consistent, the data block verification result is determined to be successful. In response to the inconsistency between the checksum and comparison results, the data block verification result is determined to be unsuccessful.

[0078] Here, the preset checksum algorithm is a pre-determined algorithm used to calculate the checksum of the data block. It remains consistent throughout the data writing and verification phases to ensure the comparability of checksums. Examples include CRC32, CRC64, Adler-32, and SHA-256.

[0079] When the real-time checksum is exactly equal to the pre-stored checksum, it indicates that the data block is not corrupted; when the real-time checksum differs from the pre-stored checksum, it indicates that the data block may be corrupted.

[0080] In some specific implementations, the method further includes: Detect the background task corresponding to the storage unit to be verified; In response to the background task being a refactoring task, the header verification result is used as the verification result of the storage unit to be verified corresponding to the background task, and the pre-verification cache is updated. In response to the background task being a garbage collection task, tail reverse verification and data block verification are performed on the storage unit to be verified corresponding to the background task. Valid data blocks of the storage unit to be verified are identified and marked, and the pre-verification cache is updated.

[0081] Specifically, for refactoring tasks: performance is prioritized by default, and standard verification is used by default; for garbage collection tasks: reliability is prioritized by default, and deep verification is used by default; for deduplication tasks: adjustments are made automatically based on the above dynamic decision-making logic.

[0082] In some specific implementations, the strategy factors may also include hot / cold factors and priority factors. Data block verification of the storage unit to be verified also includes: Based on the strategy factor, the execution strategy for data block verification is adaptively adjusted; The adjustment of the execution strategy for data block verification includes at least one of the following: In response to a load factor exceeding the first load threshold, parallel verification calculations are performed on the data blocks of the storage unit to be verified. In response to the hot / cold factor indicating that the storage unit to be verified is hot data, the metadata area data block corresponding to the header of the storage unit to be verified is verified first. In response to the priority factor indicating that the logical volume to which the storage unit to be verified belongs is of high priority, the strength of the preset verification algorithm is increased or a second sampling verification is performed.

[0083] Here, parallel verification computation is an execution strategy that simultaneously starts multiple computation threads or processes to concurrently perform checksum calculations and comparisons on multiple data blocks.

[0084] Here, the metadata area describes information about the internal data structure of the FU, such as finer-grained block allocation tables, extended attributes, etc.

[0085] Increasing the strength of the preset verification algorithm can include: upgrading from CRC32 to CRC64 to increase the number of checksum bits and reduce the false negative rate; using cryptographic hashing, such as using a portion of SHA-256 to provide extremely high collision resistance; and combining verifications by simultaneously calculating and comparing checksums from two different algorithms.

[0086] This enables dynamic optimization of the verification process, improving the security of background task execution and data reliability.

[0087] In some specific implementations, the method further includes: In response to a fully valid verification result, processing operations are performed on the storage unit to be verified. In response to a partially valid verification result, the system identifies valid and invalid data blocks, performs processing operations on valid data blocks, and does not perform processing operations on invalid data blocks. If the verification result is invalid, no processing operation will be performed on the data in the storage unit to be verified.

[0088] Among them, fully valid means that all data blocks of FU (including header, tail, and all user data blocks) pass the verification, and the data is complete and reliable; partially valid means that some data blocks in FU pass the verification, while other data blocks fail the verification, forming a state where valid data blocks and invalid data blocks coexist; invalid means that FU fails the basic verification (such as header verification or tail reverse verification), or all data blocks fail the verification, and the entire FU is unusable.

[0089] The processing operations may include garbage collection, data deduplication, data migration, data reconstruction, data compression, and encryption conversion.

[0090] Specifically, for fully valid Functional Units (FUs), normal garbage collection or deduplication operations are performed; for partially valid FUs, only the data blocks that pass internal verification are processed to achieve fine-grained reclamation of storage space; and for invalid FUs, a skip or alarm process is executed.

[0091] Thus, based on the aforementioned refined verification results (fully effective, partially effective, and ineffective), a differentiated data management strategy is implemented to achieve a secure, efficient, and waste-free data processing workflow.

[0092] In one embodiment, Figure 2 This is a schematic diagram of the FU data architecture in the embodiments of this application, such as... Figure 2 As shown, the FU data architecture in this application includes a header, a data area, and a tail.

[0093] Specifically, the header stores key metadata, including the magic number, version number, volume ID, number of data blocks, and header CRC; it also includes a list of data block checksums, storing an independent checksum for each PBA (Physical Block Address) data block.

[0094] Specifically, the data area includes N PBAs and their data. Each PBA corresponds to one data block.

[0095] Specifically, the tail storage includes extended verification information. This includes a reverse CRC / checksum, calculated based on the entire FU data.

[0096] In one embodiment, Figure 3 This is a schematic diagram of the multi-level verification process in an embodiment of this application, such as... Figure 3As shown, the multi-level verification process in this application includes: starting verification, querying the pre-verification cache, determining whether the cache hits, if a hit occurs, obtaining the cache result, and processing the result; if a miss occurs, reading the FU header, performing preliminary verification, determining whether the preliminary verification passes, if the preliminary verification fails, marking it as invalid, briefly caching, and processing the result; if the preliminary verification passes, making a dynamic verification decision, determining whether to perform deep verification; if deep verification is not performed, marking it as valid, updating the cache, and processing the result; if deep verification is performed, determining the verification result; if the verification result is fully valid, updating the cache, processing the result; if the verification result is partially valid, updating the cache, processing the result, and ending the process.

[0097] The embodiments of this application provide another storage unit verification method, which essentially involves constructing a closed-loop verification flow: cache-first query → preliminary verification and dynamic decision-making → multi-level deep verification execution → result processing and cache update. The method is as follows: S1: Verification Trigger and Cache-Priority Query Phase: This verification process is triggered when a Functionality Required (FU) needs to be processed, such as during garbage collection or background deduplication. This process first queries a global pre-verification cache implemented using a hash table. The cache key is the ID of the target FU, and the cache value includes the latest verification result for that FU (fully valid, partially valid, invalid), a bitmap recording the location of valid data blocks (grains), and a timestamp indicating whether the cache has expired. If the cache is hit and has not expired, the verification result is returned directly, and the process jumps to S4. This mechanism optimizes performance by avoiding redundant I / O and computation, significantly reducing verification overhead, especially for frequently accessed FUs.

[0098] S2: Preliminary Verification and Dynamic Decision-Making Phase: If the cache is not hit, substantive verification begins. The system first reads the header data of the target FU (the first grain of the FU) from the disk. The header contains the version number, magic number, CRC, etc. Preliminary verification includes: verifying whether the magic number matches the system's preset value to quickly identify illegal FU structures; verifying whether the version number is within the system's supported range to ensure compatibility; calculating the CRC value of the data and comparing it with the CRC value stored in the header to complete basic header verification. If the verification fails, the FU is marked as "invalid" and the process jumps to S4. If the preliminary verification passes, the subsequent dynamic decision-making phase begins. The decision-making mechanism is based on a weighted score of multiple strategy factors to determine which verification path to take. The specific implementation is as follows: Current system I / O load (load_factor): The current number of I / O (Input / Output Operations Per Second, IOPS) or bandwidth utilization of the system is divided by the preset maximum value to obtain a value between 0 and 1 (0 represents idle, 1 represents full load), which is used to indicate the severity of the current system load. FU Historical Fault Factor (fault_record, weight 0.4): This counts the cumulative number of times the FU has experienced verification anomalies or data corruption in the past, and maps the frequency of the faults to the range of 0 to 1. The higher the frequency of the faults, the larger the value, which means that deep verification is more necessary. Data service fault tolerance level (tolerance_level, weight 0.3): The level is assigned according to the data fault tolerance requirements of the business carried by the FU. The fault tolerance level is 1.0 for core financial / critical business, 0.5 for ordinary online business, and 0.2 for archived cold data. The higher the fault tolerance requirement, the more priority is given to deep verification. Based on the above strategy factors, the decision logic can be defined as: Decision Score = 0.3 × load_factor + 0.4 × fault_record + 0.3 × tolerance_level. The system will preset a decision threshold (default 0.5). If the decision score is lower than this threshold, it means that performance should be prioritized, that is, the decision result is to adopt the "standard verification" path (relying only on the header CRC, and the process ends); if the decision score is higher than the threshold, it means that reliability should be prioritized, that is, the decision result is to adopt the "deep verification" path, and enter the subsequent multi-level verification stage.

[0099] In addition, this application sets differentiated default strategies for different background tasks: Refactoring tasks: performance is prioritized by default, and standard verification is used by default; Garbage collection tasks: reliability is prioritized by default, and deep verification is used by default; Deletion tasks: automatically adjusted according to the above dynamic decision-making logic.

[0100] S3: Multi-level Deep Verification Execution Phase: This phase is the core of reliability assurance. Deep verification includes the following two levels: Tail verification: The system first reads the tail data block (the last grain) of the FU, which stores the reverse checksum (SHA-256 hash value) of the header data. During verification, the system recalculates the SHA-256 value of the current header data and compares it with the value stored in the tail. This two-way verification mechanism can effectively detect partial corruption errors in the header or tail, avoiding extreme scenarios where the header is tampered with but the CRC check passes. If the verification fails, the FU is marked as "invalid".

[0101] Distributed data verification: If the tail check passes, block-level verification is performed. The FU header contains a "block checksum list," which stores the checksum (CRC32) calculated for each data block (grain) within the FU. During verification, the system calculates the CRC32 value for each data block and compares it with the corresponding checksum in the list. The system maintains a bitmap to record the verification results, with each bit corresponding to a data grain. When the checksum comparison for a grain is successful, the corresponding bit is set to '1' (valid); otherwise, it is set to '0' (invalid). Simultaneously, when a grain verification fails, the system attempts to recover using RAID redundancy. If recovery is successful, the bit is set to '1'; otherwise, it is set to '0'. This error location and recovery mechanism effectively improves the system's fault tolerance. After all grains have been verified, if the bitmap is all '1', then FU is marked as "fully valid"; if the bitmap is all '0', then it is marked as "invalid"; if the bitmap contains both '1' and '0', then it is marked as "partially valid", and the bitmap is recorded for subsequent processing.

[0102] S4: Result Processing and Cache Update Phase: The system updates the verification results (including valid status and valid data bitmap) to the pre-verification cache and sets reasonable lifespans: Fully valid FU: 24 hours; Partially valid FU: 12 hours; Invalid FU: 1 hour. Simultaneously, when the cache capacity reaches a threshold, the system uses an LRU (Least Recently Used) eviction policy to evict the least recently accessed cache item, ensuring cache utilization. Subsequent tasks are handled differently based on the final status: For fully valid FUs, normal garbage collection or deduplication is performed, all valid data blocks are migrated, and the entire FU's space is reclaimed; for partially valid FUs, only the internally verified data blocks are processed, these valid data blocks are migrated to a new FU, and only the invalid portion's space is reclaimed, achieving fine-grained storage space reclamation; for invalid FUs, a skip or alarm process is executed, directly reclaiming the entire FU's space. In summary, the entire verification process forms a complete closed loop including perception, decision-making, execution, and feedback, improving the robustness and efficiency of the storage system.

[0103] In one embodiment, Figure 4 This is a schematic diagram of the dynamic multi-level verification processing timing in the embodiments of this application, as shown below. Figure 4 As shown, the dynamic multi-level verification processing sequence in this application includes: a background task triggers FU validity verification, the FU identifier ID is passed in, and the process sequentially passes through the cache query module, the preliminary verification module, the dynamic decision module, the multi-level verification module, and the result processing module.

[0104] Specifically, after the background task initiates a verification request, it enters the cache query module to perform a pre-verification cache query. If the cache is hit and has not expired, the verification result in the cache is directly returned to the background task, and the process ends. If the cache is not hit, or the cache has expired, the cache query module calls the preliminary verification module.

[0105] Specifically, the preliminary verification module reads the data header and performs magic number, version, and CRC checks. If the preliminary verification fails, the FU is marked as invalid and transferred to the result processing module. If the preliminary verification passes, the FU-related information is sent to the dynamic decision-making module to initiate a decision request.

[0106] Specifically, the dynamic decision-making module calculates multi-factor decision scores and compares them with thresholds. If the score is greater than the threshold, standard validation is selected, the function (FU) is marked as fully valid, and the result is sent to the result processing module. If the score is less than or equal to the threshold, deep validation is selected, and the multi-level validation module is invoked to perform multi-level deep validation.

[0107] Specifically, the multi-level verification module sequentially performs tail-side reverse verification, followed by independent data block verification and error recovery; then, it generates a valid data bitmap, marks the state of FU, and combines the verification results with the bitmap. Figure 1 The result is then output to the result processing module.

[0108] Specifically, the result processing module sets differentiated cache expiration times based on the status of the FU, updates the pre-verification cache, and executes the LRU eviction policy; then it returns the final verification result and validity bitmap to the background task, completing the entire verification process.

[0109] It should be understood that, although Figures 1-4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0111] Embodiments of this application also provide a storage unit verification device, comprising: a first processing module 501, configured to read the header information of the storage unit to be verified, perform header verification on the storage unit to be verified, and obtain a header verification result; a second processing module 502, configured to, in response to the header verification result being a pass verification, determine a decision score for the storage unit to be verified based on at least one strategy factor among the current system load, historical failure count, and business fault tolerance requirements; a third processing module 503, configured to, in response to the decision score being less than a preset decision threshold, use the header verification result as the verification result for the storage unit to be verified and update the pre-verification cache; and a fourth processing module 504, configured to, in response to the decision score being greater than or equal to the preset decision threshold, perform tail reverse verification and data block verification on the storage unit to be verified, identify and mark the valid data blocks of the storage unit to be verified, and update the pre-verification cache.

[0112] In a preferred embodiment of this application, the first processing module 501 is specifically used to: obtain the identifier of the storage unit to be verified, query the pre-verification cache, and determine whether the storage unit to be verified exists in the pre-verification cache; in response to the existence of the storage unit to be verified in the pre-verification cache, extract the timestamp corresponding to the storage unit to be verified, compare it with the current system time, and determine the lifespan; in response to the lifespan of the storage unit to be verified being less than or equal to the preset lifespan, obtain the valid data bitmap of the storage unit to be verified, determine that the cache verification result is valid, and use the information in the cache verification result as the verification result of the storage unit to be verified.

[0113] In a preferred embodiment of this application, the first processing module 501 is specifically configured to: read the header information of the storage unit to be verified, and parse it to obtain the magic number field, the version number field, and the cyclic redundancy check (CRCD) field; compare the magic number field with the valid magic number field value to obtain a first verification result; in response to the first verification result being a match, compare the version number field with the range of the valid version number field to obtain a second verification result; in response to the second verification result being a match, determine a third calculation field based on the header information, compare the third calculation field with the CRCD field to obtain a third verification result; and in response to the third verification result being a match, determine the header verification result as passed.

[0114] As a preferred implementation, in this embodiment of the application, the second processing module 502 is specifically used for: collecting the current number of input / output operations per second, normalizing the operations to obtain a load factor; mapping the historical failure count to a normalized interval based on the historical failure count of the storage unit to be verified, and determining the historical failure factor; assigning graded values ​​according to the data fault tolerance requirements of the services carried by the storage unit to be verified, and determining the fault tolerance factor; and obtaining a decision score through weighted calculation based on the load factor, historical failure factor, and fault tolerance factor.

[0115] In a preferred embodiment of this application, the fourth processing module 504 is specifically used for: reading the tail data block of the storage unit to be verified and parsing it to obtain the reverse checksum of the stored header information; reading and calculating the hash value based on the current header information of the storage unit to be verified using a preset hash algorithm to obtain the header checksum; comparing the header checksum and the reverse checksum of the header information to obtain the comparison result; and, in response to the comparison result being consistent, determining that the tail reverse verification is passed, performing data block verification on the storage unit to be verified, identifying the valid data block of the storage unit to be verified, and marking it.

[0116] In a preferred embodiment of this application, the fourth processing module 504 is further configured to: obtain a pre-stored data block checksum list from the header information of the storage unit to be verified, wherein the data block checksum list stores a pre-calculated checksum for each data block; initialize a valid data bitmap, wherein the number of bits in the valid data bitmap is the same as the total number of data blocks contained in the storage unit to be verified; traverse each data block in the storage unit to be verified, perform a verification operation on each data block, and obtain a data block verification result; in response to a data block verification result that passes verification, mark the corresponding position in the valid data bitmap as valid; and in response to the completion of the verification operation of each data block, detect the status of the valid data bitmap.

[0117] In a preferred embodiment of this application, the fourth processing module 504 is further configured to: read the content of the current data block, calculate the real-time checksum corresponding to the current data block using a preset verification algorithm; compare the real-time checksum with the pre-stored checksum to determine the checksum comparison result; in response to the checksum comparison result being consistent, determine that the data block verification result is passed; and in response to the checksum comparison result being inconsistent, determine that the data block verification result is failed.

[0118] In a preferred embodiment of this application, the device further includes a fifth processing module, which is specifically used for: detecting the background task corresponding to the storage unit to be verified; in response to the background task being a reconstruction task, using the header verification result as the verification result of the storage unit to be verified corresponding to the background task, and updating the pre-verification cache; in response to the background task being a garbage collection task, performing tail reverse verification and data block verification on the storage unit to be verified corresponding to the background task, identifying and marking the valid data blocks of the storage unit to be verified, and updating the pre-verification cache.

[0119] In a preferred embodiment of this application, the device further includes a sixth processing module, which is specifically used for: performing processing operations on the storage unit to be verified in response to a verification result that is fully valid; identifying valid data blocks and invalid data blocks in response to a verification result that is partially valid, performing processing operations on valid data blocks and not performing processing operations on invalid data blocks; and not performing processing operations on the data in the storage unit to be verified in response to a verification result that is invalid.

[0120] For a description of the features in the embodiment corresponding to the storage cell verification device, please refer to the relevant description of the embodiment corresponding to the storage cell verification method, which will not be repeated here.

[0121] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described storage cell verification method embodiments.

[0122] The electronic device can be the target device, and its internal structure diagram can be as follows: Figure 6 As shown, the electronic device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores storage cell verification data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a storage cell verification method.

[0123] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described storage unit verification method embodiments when running.

[0124] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0125] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described storage unit verification method embodiments.

[0126] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described storage cell verification method embodiments.

[0127] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] The foregoing has provided a detailed description of a storage cell verification method, electronic device, storage medium, and computer program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for verifying a storage cell, characterized in that, The method includes: Read the header information of the storage unit to be verified, parse the metadata field in the header information, verify the metadata field according to the preset verification rules, and obtain the header verification result; In response to the header verification result being verified as passed, the decision score of the storage unit to be verified is determined based on at least one of the following strategy factors: current system load, number of historical failures, and business fault tolerance requirements. In response to the decision score being less than a preset decision threshold, the header verification result is used as the verification result of the storage unit to be verified, and the pre-verification cache is updated; In response to the decision score being greater than or equal to the preset decision threshold, the tail metadata of the storage unit to be verified is read, consistency verification is performed in combination with the header information, and the validity of each data block in the storage unit to be verified that passes the verification is performed. The valid data blocks of the storage unit to be verified are identified and marked, and the pre-verification cache is updated.

2. The storage cell verification method according to claim 1, characterized in that, Before reading the header information of the storage unit to be verified, the method further includes: Obtain the identifier of the storage unit to be verified, query the pre-verification cache, and determine whether the storage unit to be verified exists in the pre-verification cache; In response to the presence of the storage unit to be verified in the pre-verification cache, the timestamp corresponding to the storage unit to be verified is extracted, compared with the current system time, and the lifespan is determined. In response to the storage unit to be verified having a lifespan less than or equal to a preset lifespan, a valid data bitmap of the storage unit to be verified is obtained, the cache verification result is determined to be valid, and the information in the cache verification result is used as the verification result of the storage unit to be verified.

3. The storage cell verification method according to claim 1, characterized in that, The process involves reading the header information of the storage unit to be verified, parsing the metadata fields in the header information, verifying the metadata fields according to preset verification rules, and obtaining the header verification result, including: Read the header information of the storage unit to be verified, and parse it to obtain the magic number field, version number field, and cyclic redundancy check code field; By comparing the magic number field with the valid magic number field value, the first verification result is obtained; In response to the first verification result being a match, the version number field and the range of valid version number fields are compared to obtain a second verification result; In response to the second verification result being a match, a third calculated field is determined based on the header information, and the third calculated field is compared with the cyclic redundancy check code field to obtain a third verification result; In response to the third verification result being a match, the header verification result is determined to be a pass verification.

4. The storage cell verification method according to claim 1, characterized in that, The decision score for the storage unit to be verified is calculated based on at least one strategy factor among the current system load, historical failure count, and business fault tolerance requirements, including: Collect the current number of input / output operations per second, normalize the number of operations, and obtain the load factor; Based on the historical failure count of the storage unit to be verified, the historical failure count is mapped to a normalized interval to determine the historical failure factor; Based on the data fault tolerance requirements of the storage unit to be verified, graded values ​​are assigned to determine the fault tolerance factor. The decision score is obtained by weighted calculation based on the load factor, the historical failure factor, and the fault tolerance factor.

5. The storage cell verification method according to claim 1, characterized in that, The process of reading the tail metadata of the storage unit to be verified, performing consistency verification in conjunction with the header information, verifying the validity of each data block in the storage unit to be verified that passes the verification, and identifying and marking the valid data blocks of the storage unit to be verified includes: Read the tail data block of the storage unit to be verified and parse it to obtain the reverse checksum of the stored header information; Read the current header information of the storage unit to be verified, calculate the hash value using a preset hash algorithm, and obtain the header checksum. The comparison result is obtained by comparing the header checksum with the header information reverse checksum. In response to the matching result being consistent, the validity of each data block in the storage unit to be verified is checked, and the valid data blocks of the storage unit to be verified are identified and marked.

6. The storage cell verification method according to claim 5, characterized in that, The step of validating each data block in the storage unit to be verified, identifying and marking the valid data blocks in the storage unit to be verified, includes: From the header information of the storage unit to be verified, obtain a pre-stored checksum list of data blocks, wherein the checksum list stores a pre-calculated checksum for each data block; Initialize a valid data bitmap, wherein the number of bits in the valid data bitmap is the same as the total number of data blocks contained in the storage unit to be verified; Traverse each data block in the storage unit to be verified, perform a verification operation on each data block, and obtain the data block verification result; In response to the data block verification result being successful, the corresponding position in the valid data bitmap is marked as valid; After the verification operation of each data block is completed, the status of the valid data bitmap is checked.

7. The storage cell verification method according to claim 6, characterized in that, The step of performing verification operations on each data block to obtain data block verification results includes: Read the contents of the current data block and calculate the real-time checksum corresponding to the current data block using a preset checksum algorithm; Compare the real-time checksum with the pre-stored checksum to determine the checksum comparison result; In response to the fact that the verification and comparison results are consistent, the data block verification result is determined to be passed. In response to the inconsistency of the verification and comparison results, the data block verification result is determined to be unsuccessful.

8. The memory cell verification method according to any one of claims 1-7, characterized in that, The method further includes: Detect the background task corresponding to the storage unit to be verified; In response to the background task being a reconstruction task, the header verification result is used as the verification result of the storage unit to be verified corresponding to the background task, and the pre-verification cache is updated. In response to the background task being a garbage collection task, tail reverse verification and data block verification are performed on the storage unit to be verified corresponding to the background task, valid data blocks of the storage unit to be verified are identified and marked, and the pre-verification cache is updated.

9. The storage cell verification method according to claim 1, characterized in that, The method further includes: In response to the verification result being fully valid, a processing operation is performed on the storage unit to be verified. In response to the verification result being partially valid, valid data blocks and invalid data blocks are identified. The processing operation is performed on the valid data blocks, and the processing operation is not performed on the invalid data blocks. In response to the invalid verification result, the processing operation is not performed on the data of the storage unit to be verified.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the storage cell verification method as described in any one of claims 1-9 when executing the computer program.

Images