Electronic archive data processing method and device

Abstract

The invention provides an electronic archive data processing method and device. The method comprises the following steps: performing content segmentation on to-be-processed electronic archive data according to content characteristics of the to-be-processed electronic archive data to obtain a plurality of data blocks; generating a corresponding hash value for each data block by adopting a hash algorithm; inquiring whether a hash value exists in a global fingerprint index database or not; if so, establishing a logic reference relationship pointing to the existing data block; if the hash value does not exist, storing the data block corresponding to the hash value, and registering the hash value into the global fingerprint index database; and on the basis of the hash values of the plurality of data blocks, constructing a Merkle tree-shaped data structure taking the hash values as nodes, and generating a root hash representing the complete state of the file through recursive calculation. According to the method, an endogenous and efficient data representation and integrity verification mechanism is established for the electronic archive through a core method flow of blocking, duplicate removal and Merkle tree construction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage and verification, in particular to an electronic archive data processing method and device. BACKGROUND

[0002] With the deepening of government informationization and enterprise digital transformation, the data volume of electronic archives is growing explosively. Such data has the characteristics of high data redundancy (such as the use of the same template for a large number of official documents, and the existence of multiple versions of the same archive with minor differences), strict data consistency requirements (to meet the compliance audit of off-site backup), and high risk of long-term preservation (threatened by disk silent errors or human tampering).

[0003] Currently, the industry mainly uses the following technical solutions for the storage, backup and verification of electronic archives: Traditional full / incremental backup technology: files are backed up by being packed periodically. This technology is inefficient when dealing with minor modifications, and even if only one word is modified, the incremental backup usually needs to backup the entire file or changes based on the file system, which cannot achieve global content deduplication across files, resulting in waste of storage space and network bandwidth.

[0004] Rsync synchronization technology: the checksum of the file block is compared to transfer the difference. This technology has large calculation overhead and requires a large number of I / O operations at the source and destination, usually in a point-to-point mode, lacking global version management and deduplication index, making it difficult to efficiently support massive small file or super large file synchronization scenarios.

[0005] Traditional RAID and file-level checksum technology (such as MD5): use hardware RAID to prevent disk damage, or perform integrity verification by calculating the MD5 value of the file. However, the MD5 algorithm has a collision risk, and it takes a very long time to perform full-disk scanning and verification on massive archives, making it difficult to run in real time. More importantly, these technologies cannot effectively prevent file content from being tampered with maliciously, and lack endogenous consistency verification and self-repair capabilities.

[0006] In summary, the main technical problems existing in the current field can be summarized as follows: the existing electronic archive storage system has the problems of serious waste of storage space, high off-site synchronization bandwidth occupation, low consistency verification efficiency of massive data, and lack of endogenous, efficient tamper-proofing and long-term preservation guarantee mechanism when dealing with massive, high-redundancy data. SUMMARY

[0007] The purpose of the embodiments of the present application is to provide an electronic archive data processing method and device, which establishes a basic data processing method for electronic archives with high storage efficiency, efficient verification capability and high synchronization potential through the core process of content block-deduplication storage-verification structure construction.

[0008] In a first aspect, a method for processing electronic archive data is provided, which can include: segmenting the electronic archive data to be processed according to its content features to obtain a plurality of data blocks; generating a corresponding hash value for each data block using a hash algorithm, and querying whether the hash value exists in a global fingerprint index library; if it exists, establishing a logical reference relationship pointing to the existing data block; if it does not exist, storing the data block corresponding to the hash value and registering the hash value in the global fingerprint index library; based on the hash values of the plurality of data blocks, constructing a Merkle tree-shaped data structure with hash values as nodes, and generating a root hash representing the complete state of the archive through recursive calculation.

[0009] In one possible implementation, segmenting the electronic archive data to be processed according to its content features to obtain a plurality of data blocks includes: reading the electronic archive data to be processed in the form of a data stream; creating a fixed-length sliding window at the starting position of the data stream; calculating the hash value of the data in the current sliding window as a rolling hash value; determining whether the rolling hash value meets a preset splitting condition; if it does, determining the boundary position of the current sliding window as a data splitting point, and taking the data from the previous splitting point or the starting position to the current splitting point as a data block; if it does not, moving the sliding window one predetermined step length in the direction of the data stream to obtain a new current sliding window, and returning to the step of calculating the hash value of the data in the current sliding window as a rolling hash value until the data stream is completely traversed to obtain a plurality of data blocks corresponding to the electronic archive data to be processed.

[0010] In one possible implementation, the preset splitting condition is that the remainder of the rolling hash value modulo a preset value is equal to a specific value.

[0011] In one possible implementation, the logical reference relationship is maintained through a configured file recipe, and the file recipe includes an ordered list of hash values of the data blocks.

[0012] In one possible implementation, when processing a new version of the electronic archive data, the method further includes: segmenting the new version of the electronic archive data according to its content features to obtain a plurality of data blocks of the new version; identifying coincident data blocks in the plurality of data blocks of the new version with the plurality of data blocks of the electronic archival data to be processed; for the coincident data blocks in the plurality of data blocks of the new version, directly referencing their hash values to the corresponding hash values in the Merkle tree data structure of the electronic archival data to be processed; for the non-coincident data blocks in the plurality of data blocks of the new version, calculating their new hash values; updating the Merkle tree data structure based on the new hash values to generate a root hash of the new version, wherein the updated Merkle tree data structure shares nodes with unchanged hash values with the pre-updated Merkle tree data structure.

[0013] In one possible implementation, the method further comprises: when it is necessary to verify the integrity of the archive, reading the data blocks corresponding to the stored electronic archival data and the corresponding stored root hash; recomputing the current hash value of each data block; reconstructing the Merkle tree data structure based on the recomputed hash values and obtaining the corresponding current root hash; comparing the current root hash with the stored root hash, and determining that the archival data is abnormal if they are inconsistent.

[0014] In one possible implementation, the method further comprises: traversing each data block; reading any data block and recomputing its current hash value; comparing the recomputed current hash value with the registered hash value in the global fingerprint index library; if the comparison is inconsistent, marking the data block as damaged and automatically reconstructing the damaged data block from the redundant storage node or using the error correction code check block.

[0015] In a second aspect, a processing device for electronic archival data is provided, which can comprise: a segmentation unit configured to content segment the electronic archival data to be processed according to its content features to obtain a plurality of data blocks; a generation unit configured to generate a corresponding hash value for each data block using a hash algorithm; a query unit configured to query whether the hash value exists in a global fingerprint index library; a building unit configured to, if the hash value exists, establish a logical reference relationship pointing to the existing data block; a storage unit configured to, if the hash value does not exist, store the data block corresponding to the hash value and register the hash value in the global fingerprint index library; a building unit configured to build a Merkle tree data structure with hash values as nodes based on the hash values of the plurality of data blocks; The generation unit is further configured to generate a root hash representing a complete state of the archive by recursive calculation.

[0016] In a third aspect, an electronic device is provided, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method steps of any one of the first aspect.

[0017] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the first aspect.

[0018] Embodiments of the present application provide an electronic archive data processing method and device, which divides the electronic archive data to be processed according to its content features to obtain a plurality of data blocks, generates corresponding hash values for each data block using a hash algorithm, queries whether the hash values exist in a global fingerprint index library, establishes a logical reference relationship pointing to the existing data block if the hash values exist, stores the data block corresponding to the hash value and registers the hash value in the global fingerprint index library if the hash values do not exist, constructs a Merkle tree-shaped data structure with hash values as nodes based on the hash values of the plurality of data blocks, and generates a root hash representing a complete state of the archive by recursive calculation. The method establishes an endogenous and efficient data representation and integrity verification mechanism for electronic archives through the core method flow of block-de-duplication-Merkle tree construction, and lays a structural foundation for subsequent storage optimization, rapid synchronization and security verification. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0020] Figure 1 A flowchart of an electronic archive data processing method provided by an embodiment of the present application; Figure 2 A structural diagram of an electronic archive data processing device provided by an embodiment of the present application; Figure 3 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, and not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the general meaning understood by those of ordinary skill in the art to which the present application belongs. The terms "first", "second", and similar terms used in the present application do not represent any order, number, or importance, but are only used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect", "couple", or "connect" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0022] The preferred embodiments of the present application are described below in conjunction with the drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0023] Figure 1 A flowchart of an electronic archive data processing method provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method can include the following steps. Figure 1 Step S110, the electronic archive data to be processed is content segmented according to its content characteristics, to obtain a plurality of data blocks.

[0024] In a specific implementation, the electronic archive data to be processed is read in the form of a data stream; wherein the electronic archive includes multiple types such as OFD, PDF, streaming media, database backup, and does not depend on the physical path identifier of the traditional file system.

[0025] At the starting position of the data stream, a fixed-length sliding window is created; the hash value of the data in the current sliding window is calculated as a rolling hash value; it is judged whether the rolling hash value satisfies the preset splitting condition, and the preset splitting condition is that the remainder of the quotient of the modulus of the rolling hash value and the preset value is equal to a specific value.

[0026] ​If the conditions are met, the boundary position of the current sliding window is determined as the data splitting point, and the data from the previous splitting point or the starting position to the current splitting point is taken as a data block; if the conditions are not met, the sliding window is moved backward by a predetermined step along the direction of the data flow to obtain a new current sliding window, and the execution step is returned: calculate the hash value of the data in the current sliding window as the rolling hash value, until the entire data flow is traversed to obtain multiple data blocks corresponding to the electronic archive data to be processed.

[0027] Specifically, a fixed-length sliding window is created at the beginning of the data stream, with the window size configured to 48 bytes (which can be adjusted according to the actual application scenario, such as 32 bytes or 64 bytes), and the Rabin Fingerprint algorithm or BuzzHash algorithm is used as the rolling hash function.

[0028] Traverse the data stream byte by byte and calculate the hash value of the data in the current sliding window in real time, which is used as the scrolling hash value F.

[0029] The default splitting condition is that the remainder after taking the rolling hash value F modulo the average block size D equals a specific value R (i.e., F mod D = R), where D and R can be dynamically configured according to the target block size. When this splitting condition is met, the boundary position of the current sliding window is determined as the data splitting point, and the data from the previous splitting point (or the start position of the data stream) to the current splitting point is treated as an independent data block; if the splitting condition is not met, the sliding window is moved backward by 1 byte along the data stream direction (the predetermined step size can be configured to 1-4 bytes), and the rolling hash value is recalculated until the entire data stream has been traversed.

[0030] After setting the minimum capacity threshold of 2KB and the maximum capacity threshold of 64KB for data blocks (which can be adjusted according to the file type), if the candidate data blocks obtained according to the splitting conditions are smaller than the minimum threshold, they are forcibly merged into the next data block; if they are larger than the maximum threshold, a splitting point is forcibly set at the maximum threshold position to avoid generating data blocks that are too small, resulting in excessive metadata overhead, or too large, resulting in reduced deduplication rate.

[0031] This step implements fine-grained variable-length blocks based on the content characteristics of electronic archives, solving the problem of deduplication failure of fixed blocks during data insertion / deletion operations.

[0032] Step S120: Use a hash algorithm to generate corresponding hash values ​​for each data block.

[0033] In practice, the SHA-256 algorithm or the national cryptographic SM3 algorithm (with support for algorithm switching configuration) is used to perform hash calculations on each data block. The hash algorithm has the characteristics of strong anti-collision and high computational efficiency, which can ensure that the hash value of different data blocks is globally unique.

[0034] The calculated hash value is used as the content identifier (CID) of the corresponding data block. This identifier is not only used for the unique identification of the data block, but also serves as the core basis for subsequent storage addressing, deduplication judgment and integrity verification.

[0035] Step S130: Query the global fingerprint index library to see if a corresponding hash value exists, and maintain the global fingerprint index library based on the query results.

[0036] Build a global fingerprint index library and store it using a high-performance key-value database (such as RocksDB or Redis). The key of the index is the CID (hash value) of the data block, and the value includes the physical storage location of the data block (such as object storage Bucket + ObjectKey + Offset) and the reference count, which is used to record how many files or versions reference the data block.

[0037] Construct a file recipe / manifest as a logical layer index. The key is the file ID or version number, and the value is an ordered list of CIDs ([Hash_1,Hash_2,...,Hash_n]). This is used to record the logical structure of the file and decouple the data entity from the logical structure.

[0038] Before querying the global fingerprint index, the CID of the data block is first input into a Bloom filter for preliminary judgment. The Bloom filter pre-caches the CID features of the stored data blocks. If it is determined that the data block definitely does not exist, the data block storage operation is performed directly; if it is determined that the data block may exist, the global fingerprint index query process is initiated. This reduces the query pressure on the KV database and improves the efficiency of deduplication.

[0039] If the CID exists, a logical reference relationship is established pointing to the existing data block. This logical reference relationship is maintained through the configured file recipe, which includes an ordered list of hash values ​​for each data block. Specifically, if the CID exists in the global fingerprint index, it means that the data block has been stored in other archives or versions. In this case, the physical data block is not rewritten; instead, the CID is added to the recipe of the current file, and the reference count corresponding to the CID in the global fingerprint index is incremented by 1, achieving "zero-copy" deduplication.

[0040] If the hash value does not exist, the corresponding data block is stored, and the hash value is registered in the global fingerprint index. Specifically, if the CID does not exist in the global fingerprint index, the data block is compressed (using GZIP or LZ4 compression algorithm) and encrypted (using AES-256 encryption algorithm), then written to the underlying object storage (such as compatible storage, distributed file system), and the physical storage address of the data block is obtained; subsequently, the CID and its physical address are registered in the global fingerprint index, the reference count is set to 1, and the CID is added to the recipe of the current file.

[0041] Step S140: Based on the hash values ​​of multiple data blocks, construct a Merkle tree data structure with hash values ​​as nodes, and generate a root hash representing the complete state of the archive through recursive calculation.

[0042] In practice, the CID of each data block is used as the leaf node of the Merkle tree. The leaf nodes are grouped into groups of N (e.g., 128) (configurable to 64 or 256), and the combined hash value of the CID of each group of child nodes is calculated (using the SHA-256 algorithm, the combination method is to concatenate the CID of child nodes in order and then calculate the hash), which is used as the parent node. This combined hash calculation process is executed recursively until a unique root hash (Merkle Root) is generated. The root hash uniquely represents the complete state of the entire electronic archive.

[0043] Subsequently, the business metadata of the electronic archives (including title, generation time, author, archive number, etc.) is serialized into binary data blocks, which participate in the construction of the Merkle tree as a special "metadata block". That is, the CID of the metadata block is one of the leaf nodes of the Merkle tree, so that the root hash is bound to both the content and metadata of the archive. Any tampering with either side will cause the root hash to change.

[0044] The completed Merkle tree data structure (including the CID of each node and the hierarchical relationship) is stored together with the file recipe and root hash as archive metadata. The metadata is associated with the corresponding physical data block through CID, realizing the "content is address" addressing mode.

[0045] In some embodiments, when processing new versions of electronic archival data (such as draft revisions, addition of signature pages, metadata changes), this application may perform version management steps, including: The new version of electronic archive data is segmented according to its content characteristics to obtain multiple data blocks of the new version; Identify overlapping data blocks (i.e., data blocks whose content has not changed) and non-overlapping data blocks (i.e., newly added or modified data blocks) with multiple data blocks of the electronic archive data to be processed (old version data) in the new version; among them, non-overlapping data blocks include newly added data blocks and modified data blocks that modify the original data blocks.

[0046] For overlapping data blocks, their hash values ​​are directly referenced to the corresponding hash values ​​of the electronic archive data to be processed in the Merkle tree data structure. Specifically, for overlapping data blocks, there is no need to recalculate the CID and store physical data. The CID corresponding to the old version is directly referenced in the new version of the file recipe, and the reference count of the CID in the global fingerprint index is incremented by 1 to achieve sharing between the old and new version data blocks.

[0047] For non-overlapping data blocks (data blocks that have changed), calculate their new hash value, register the new CID in the global fingerprint index, or update the physical address and reference count of the original CID.

[0048] The Merkle tree data structure is updated based on the new hash value to generate a new version of the root hash. Specifically, the Merkle tree is reconstructed based on the new set of CIDs (including reused old CIDs and newly added CIDs).

[0049] Since only the CIDs of non-overlapping data blocks change, only the hash values ​​of the corresponding parent nodes and higher levels need to be recalculated. The unchanged subtree nodes directly reuse the old version's hash values, ultimately generating the new version's root hash. The new version's metadata only stores the new root hash, file recipe, and version identifier; it does not need to store the full dataset, achieving incremental version management. In other words, the updated Merkle tree data structure shares nodes with unchanged hash values ​​with the previous Merkle tree data structure.

[0050] In some embodiments, when it is necessary to verify the integrity of a file, this application may perform an integrity verification step, including: Read the data blocks corresponding to the stored electronic archive data and the corresponding stored root hash; Recalculate the current hash value for each data block; The Merkle tree data structure is reconstructed based on the recalculated hash value, and the corresponding current root hash is obtained; The current root hash is compared with the stored root hash. If they match, the file data is determined to be complete and unaltered. If they do not match, the file data is determined to be abnormal (tampered with or corrupted), triggering an alarm mechanism.

[0051] It should be noted that while reading the data blocks corresponding to the stored electronic archive data and the corresponding stored root hash, the corresponding file recipe can also be read. In addition, during the process of reading data blocks, a streaming computing method is used to verify the CID in real time. If the current CID of a certain data block is inconsistent with the record in the file recipe, the reading is immediately interrupted and an error is reported, without having to traverse all data blocks, thus improving the verification efficiency.

[0052] Furthermore, when it is necessary to prove the integrity and ownership of a specific data block in the archive (such as a page or clause of a contract) to a third party (such as a court or auditing institution), the content of that specific data block and its path information in the Merkle tree can be extracted. Path information may include: the CID (leaf node) of the specific data block, and the CIDs (i.e., Merkle Path) of all sibling nodes on the path from the leaf node to the root node.

[0053] After receiving the content of a specific data block, the Merkle Path, and the original root hash, the third party recursively calculates the hash of the parent node, starting from the leaf node, according to the hash calculation rules of the Merkle tree, until the final root hash is generated. If the generated root hash matches the original root hash, it proves that the specific data block belongs to the original file and has not been tampered with, achieving lightweight evidence without disclosing the entire file.

[0054] In some embodiments, to address silent errors such as "bit rotation" caused by the aging of physical storage media and to ensure the long-term availability of electronic files, this application may also perform data inspection and self-healing steps, including: First, configure the inspection process: run the "Data Scrubbing" process, which uses a strategy that combines random sampling and periodic traversal. Prioritize scanning buckets with longer storage times and lower access frequencies to ensure that all stored data blocks are covered within a 3-month period (configurable to 1-6 months).

[0055] Secondly, during the inspection process, each data block is traversed, and the currently traversed data block is read and its current hash value is recalculated; The recalculated current hash value is compared with the hash value registered in the global fingerprint index. If the comparison matches, the data block is determined to be normal, and the inspection continues to the next data block.

[0056] If the comparison is inconsistent, the data block is marked as corrupted, and the corrupted data block is automatically reconstructed from the redundant storage node or using erasure coding check block. The location and related information of the corrupted data block are recorded, and an alarm is triggered to notify the administrator.

[0057] If erasure coding is used for storage, the damaged block is reconstructed using the remaining data blocks and check blocks. If multiple replica storage is used, data is copied from other healthy replica nodes to overwrite the damaged block.

[0058] Furthermore, if the local repair is not possible, a request is sent to the off-site disaster recovery node based on the root hash of the index record to download the correct data block for replacement.

[0059] In some embodiments, for disaster recovery architectures such as "two sites and three centers," to achieve low-bandwidth, high-efficiency consistency synchronization between the primary storage node and the standby storage node, this application can perform cross-database synchronization steps, including: When a cross-database synchronization operation is triggered (such as scheduled synchronization, manual triggering, or triggering after data update), the primary storage node (Archive A) and the backup storage node (Archive B) establish a communication connection, and through the established communication connection, the primary node sends the original root hash of the target electronic archive to the backup node.

[0060] After receiving the original root hash of the target electronic document data, the backup storage node queries the metadata of the target electronic document data stored locally: If the same root hash exists locally, it proves that the data in the two locations are completely consistent, and the synchronization ends. The entire process only transmits tens of bytes of root hash, with no additional data transmission.

[0061] If the local root hash does not exist or is inconsistent, the standby storage node sends a request to the primary storage node to obtain the list of CIDs of the first-level child nodes of the Merkle tree corresponding to the original root hash.

[0062] The backup storage node compares the list of first-level child node CIDs in its local Merkle tree with the list sent by the primary storage node to identify missing or inconsistent child node CIDs: if the child node is a leaf node (corresponding to a specific data block), it is marked as a differential data block.

[0063] If the child node is not a leaf node, the next level child node CID list is requested from the primary storage node. The comparison process is repeated until all the different data blocks (leaf nodes) are located. The time complexity of this process is O(logN) (N is the total number of data blocks).

[0064] Difference block transmission and synchronization: The primary storage node only compresses and encrypts the missing difference data blocks from the secondary storage node before transmitting them to the secondary storage node, supporting breakpoint resumption and out-of-order transmission. If the network is interrupted during transmission, the root hash comparison and difference location process will be re-executed during the next synchronization, without transmitting data blocks that have already been synchronized. After receiving the difference data blocks, the secondary storage node stores the data blocks, updates the global fingerprint index, and reconstructs the Merkle tree to generate a root hash consistent with that of the primary storage node, thus completing the synchronization.

[0065] In some embodiments, after generating the Merkle tree, the computer program not only records the root hash but also extracts the tree's topological features (such as the tree's depth, the sequence of child nodes for each non-leaf node, and the distribution of leaf nodes), and calculates a fuzzy hash describing the topology, called the structural fingerprint. An inverted index is then created between the structural fingerprint and the archive's root hash list.

[0066] When a user needs to find other files that are structurally similar to a certain file (such as a drawing or contract template), the structural fingerprint of the target file's Merkle tree is calculated, and files with the same or similar structural fingerprints are searched in the inverted index, returning the results quickly without scanning all the content.

[0067] The electronic archive data processing method provided in this application achieves ultimate optimization of storage space and network bandwidth and synergistic improvement of inherent security and reliability by integrating content-addressed storage and Merkle tree structure in the storage, synchronization and verification of electronic archive data.

[0068] Corresponding to the above method, embodiments of this application also provide an electronic archive data processing device, such as... Figure 2 As shown, the device includes: The segmentation unit 210 is used to segment the electronic archive data to be processed according to its content characteristics to obtain multiple data blocks; The generation unit 220 is used to generate corresponding hash values ​​for each data block using a hash algorithm; Query unit 230 is used to query whether the hash value exists in the global fingerprint index library; Establishment unit 240 is used to establish a logical reference relationship pointing to an existing data block if it exists; Storage unit 250 is used to store the data block corresponding to the hash value if it does not exist, and to register the hash value to the global fingerprint index library; Construction unit 260 is used to construct a Merkle tree data structure with hash values ​​as nodes based on the hash values ​​of the plurality of data blocks; The generation unit 220 is also used to generate a root hash representing the complete state of the archive through recursive computation.

[0069] The functions of each functional unit of the electronic archive data processing device provided in the above embodiments of this application can be implemented through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the electronic archive data processing device provided in the embodiments of this application will not be repeated here.

[0070] This application also provides an electronic device, such as... Figure 3 As shown, it includes a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340.

[0071] Memory 330 is used to store computer programs; When the processor 310 executes the program stored in the memory 330, it performs the following steps: The electronic archive data to be processed is segmented according to its content characteristics to obtain multiple data blocks; A hash algorithm is used to generate a corresponding hash value for each data block; and the existence of the hash value is checked in the global fingerprint index. If it exists, a logical reference relationship is established pointing to the existing data block; if it does not exist, the data block corresponding to the hash value is stored, and the hash value is registered in the global fingerprint index library. Based on the hash values ​​of the multiple data blocks, a Merkle tree data structure with hash values ​​as nodes is constructed, and a root hash representing the complete state of the archive is generated through recursive calculation.

[0072] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0073] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0074] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0075] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0076] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 1 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0077] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform the electronic archive data processing method described in any of the above embodiments.

[0078] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the electronic archive data processing method described in any of the above embodiments.

[0079] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0084] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims in this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for processing electronic archival data, characterized in that, The method includes: The electronic archive data to be processed is segmented according to its content characteristics to obtain multiple data blocks; A hash algorithm is used to generate a corresponding hash value for each data block; and the existence of the hash value is checked in the global fingerprint index. If it exists, a logical reference relationship is established pointing to the existing data block; if it does not exist, the data block corresponding to the hash value is stored, and the hash value is registered in the global fingerprint index library. Based on the hash values ​​of the multiple data blocks, a Merkle tree data structure with hash values ​​as nodes is constructed, and a root hash representing the complete state of the archive is generated through recursive calculation.

2. The method as described in claim 1, characterized in that, The electronic archival data to be processed is segmented according to its content characteristics, resulting in multiple data blocks, including: The electronic archive data to be processed is read in the form of a data stream; At the beginning of the data stream, create a sliding window of fixed length; Calculate the hash value of the data within the current sliding window, and use it as the scroll hash value; Determine whether the rolling hash value meets the preset splitting conditions; If the conditions are met, the boundary position of the current sliding window is determined as the data splitting point, and the data from the previous splitting point or the starting position to the current splitting point is treated as a data block. If the conditions are not met, the sliding window is moved backward by a predetermined step along the direction of the data flow to obtain a new current sliding window, and the execution steps are returned: calculate the hash value of the data in the current sliding window as the rolling hash value, until the entire data flow is traversed to obtain multiple data blocks corresponding to the electronic archive data to be processed.

3. The method as described in claim 2, characterized in that, The preset segmentation condition is that the remainder after taking the rolling hash value modulo a preset value is equal to a specific value.

4. The method as described in claim 1, characterized in that, The logical reference relationships are maintained through a configured file recipe, which includes an ordered list of the hash values ​​of each data block.

5. The method as described in claim 1, characterized in that, When processing a new version of the electronic record data, the method further includes: The new version of electronic archive data is segmented according to its content characteristics to obtain multiple data blocks of the new version; Identify overlapping data blocks in multiple data blocks of the new version that are also data blocks of the electronic archive data to be processed; For overlapping data blocks in multiple data blocks in the new version, their hash values ​​are directly referenced to the corresponding hash values ​​of the electronic archive data to be processed in the Merkle tree data structure. For non-overlapping data blocks among multiple data blocks in the new version, calculate their new hash values; The Merkle tree data structure is updated based on the new hash value to generate a new version of the root hash, wherein the updated Merkle tree data structure shares nodes with unchanged hash values ​​with the Merkle tree data structure before the update.

6. The method as described in claim 1, characterized in that, The method further includes: When it is necessary to verify the integrity of the archive, read the data block corresponding to the stored electronic archive data and the corresponding stored root hash; Recalculate the current hash value for each data block; The Merkle tree data structure is reconstructed based on the recalculated hash value, and the corresponding current root hash is obtained; The current root hash is compared with the stored root hash. If they do not match, the file data is determined to be abnormal.

7. The method as described in claim 1, characterized in that, The method further includes: Iterate through each data block; Read any data block and recalculate its current hash value; The recalculated current hash value is compared with the hash value registered in the global fingerprint index. If the comparison is inconsistent, the data block is marked as corrupted, and the corrupted data block is automatically reconstructed from redundant storage nodes or using erasure coding check blocks.

8. A device for processing electronic archival data, characterized in that, The device includes: The segmentation unit is used to divide the electronic archive data to be processed into multiple data blocks according to its content characteristics. The generation unit is used to generate corresponding hash values ​​for each data block using a hash algorithm. The query unit is used to query whether the hash value exists in the global fingerprint index. Establish a unit, which, if it exists, establishes a logical reference relationship pointing to an existing data block; A storage unit is used to store the data block corresponding to the hash value if it does not exist, and to register the hash value to the global fingerprint index library; A construction unit is used to construct a Merkle tree data structure with hash values ​​as nodes based on the hash values ​​of the multiple data blocks; The generation unit is also used to generate a root hash representing the complete state of the file through recursive calculation.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images