A distributed data sharding security storage and verification method

Abstract

This invention relates to a distributed data sharding secure storage and verification method, comprising steps such as data preprocessing and sensitivity identification, dynamic adaptive sharding generation, dual encryption of shards, secure storage of encrypted shards, and hierarchical dynamic verification. This invention proposes a dynamic adaptive sharding strategy, dynamically adjusting the sharding granularity, quantity, and mapping relationship based on data sensitivity levels and node load, thus solving problems such as rigidity, data skew, and node overload in existing sharding strategies. It introduces a dual encryption mechanism, employing differentiated encryption strategies for data with different sensitivity levels, and combines this with the Shamir secret sharing algorithm to achieve distributed key management, avoiding single-point key leakage and overcoming the limitations of existing single encryption and centralized key management. Furthermore, it constructs a hierarchical dynamic verification system, combining real-time verification, periodic verification, and anomaly verification, and using the Merkle tree verification algorithm to locate tampering, overcoming the shortcomings of existing verification mechanisms such as single verification method, poor fault tolerance, and inability to locate tampering positions.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, specifically to a method for secure storage and verification of distributed data shards. Background Technology

[0002] With the rapid development of big data and cloud computing technologies, data volume is growing exponentially. Traditional centralized storage can no longer meet the storage, expansion, and high availability requirements of massive amounts of data. Distributed storage, with its advantages of strong scalability, high fault tolerance, and outstanding cost-effectiveness, has become the mainstream solution for massive data storage. Data sharding, as the core technology of distributed storage, divides massive amounts of data into multiple independent shards and stores them on different nodes, realizing distributed management and parallel access of data, effectively breaking through the storage capacity and performance bottlenecks of a single node.

[0003] However, existing distributed data sharding storage and verification methods still have many shortcomings, making it difficult to balance security, practicality, and efficiency. Specifically:

[0004] 1. Fixed sharding strategies and insufficient security: Existing technologies mostly adopt fixed hash sharding, range sharding, or consistent hash sharding strategies. The sharding rules are fixed and predictable. Attackers can analyze the sharding patterns to target specific nodes to steal or tamper with sharded data. At the same time, there is a lack of targeted encryption processing for sensitive data during the sharding process, which can easily lead to data leakage. Although some solutions use encrypted storage, the keys are mostly centrally managed, which poses a single point of leakage risk. Moreover, the binding of keys to sharded data is not implemented, making it difficult to guarantee the confidentiality of data.

[0005] 2. The verification mechanism is simple and lacks fault tolerance and accuracy: Most existing verification methods only use a single hash verification or replica verification. Hash verification is susceptible to collision attacks and cannot locate the tampering location. Although replica verification can improve fault tolerance, it requires a lot of storage space and data inconsistency problems are prone to occur during replica synchronization. At the same time, it lacks full-process verification of the integrity and consistency of the shards, making it difficult to detect abnormalities such as data tampering, loss or damage in a timely manner.

[0006] 3. Poor dynamic adaptability and difficulty in balancing performance and security: Existing sharding solutions cannot dynamically adjust sharding allocation strategies based on node load, data popularity, network status, etc., which can easily lead to problems such as data skew and node overload, affecting storage performance. At the same time, when nodes fail, expand or shrink, there is a lack of security protection during sharding migration, which can easily lead to data leakage or damage, and the migration efficiency is low, affecting system availability.

[0007] 4. Lack of traceability and auditing capabilities: Existing methods do not record and audit the entire lifecycle of data sharding operations, such as generation, storage, migration, and verification. When data security issues occur, it is impossible to trace the root cause of the problem, making it difficult to meet the compliance requirements of fields such as finance and government.

[0008] In view of the shortcomings of the existing technologies, there is an urgent need for a distributed data sharding secure storage and verification method. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this invention provides a distributed data sharding secure storage and verification method. Through innovative dynamic adaptive sharding strategies, dual encryption mechanisms, hierarchical verification systems, and full lifecycle auditing mechanisms, it achieves secure storage, accurate verification, dynamic adaptation, and traceable management of data shards, balancing security, practicality, and efficiency. It meets the massive data storage and security needs of different scenarios and solves problems such as low sharding security, inaccurate verification, poor dynamic adaptability, and lack of traceability in existing technologies.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a distributed data sharding secure storage and verification method, the method comprising the following steps:

[0011] Step S1: Data Preprocessing and Sensitivity Identification: The management node receives the data to be stored, performs data deduplication, format standardization, and redundancy compression; and classifies the preprocessed data into sensitivity levels, namely ordinary data, sensitive data, and core sensitive data, using a sensitive data identification algorithm.

[0012] Step S2: Dynamic Adaptive Sharding Generation: The management node generates a dynamic sharding strategy based on the data sensitivity level, data size, storage node load, and network bandwidth using an improved consistent hashing algorithm. The data is sharded, each shard is assigned a unique shard ID, and the shards are mapped to different available storage nodes.

[0013] Step S3: Fragmented Dual Encryption Processing: Differentiated encryption strategies are adopted according to the data sensitivity level. Core sensitive data is encrypted with "symmetric encryption + asymmetric encryption" dual encryption, sensitive data is encrypted with symmetric encryption and the key is distributed and stored through the Shamir secret sharing algorithm, and ordinary data is encrypted with lightweight symmetric encryption to generate encrypted fragments.

[0014] Step S4: Secure storage of encrypted fragments: The management node sends the encrypted fragments to the corresponding storage nodes. After the storage nodes perform a preliminary integrity check, they store the fragments in the local encrypted storage area. At the same time, backup fragments are generated for the core sensitive data fragments and stored in storage nodes in different fault domains.

[0015] Step S5: Layered Dynamic Verification: Audit nodes and management nodes collaborate to perform real-time verification, periodic verification, and anomaly verification on encrypted fragments. The Merkle tree verification algorithm is used to locate tampering and ensure data integrity and consistency.

[0016] Step S6: Shard Migration and Recovery: When a storage node fails, is overloaded, or the cluster is expanded or scaled down, the management node triggers shard migration. The migration process uses encrypted transmission and real-time verification. When a shard is lost or damaged, data is restored from the backup node and verified.

[0017] Step S7: Full lifecycle audit and traceability: Audit nodes record the full lifecycle operations of the shards, and use blockchain technology to store key audit records, so as to achieve traceability of operations and immutability of records.

[0018] Furthermore, in step S1, the sensitive data identification algorithm uses a deep learning model, combining data feature extraction and keyword matching, to automatically identify sensitive fields and classify sensitivity levels without manual intervention.

[0019] Furthermore, in step S2, the improved consistent hashing algorithm introduces a dynamic adjustment mechanism for virtual nodes, which increases or decreases the number of virtual nodes in real time according to the storage node load, to ensure that the load of each storage node is balanced and the difference in the shard storage of each node is ≤ a preset difference threshold.

[0020] Furthermore, in step S2, the granularity and number of fragments are determined according to the sensitivity level: the size of core sensitive data fragments is 16-64MB, and the number of fragments is ≥3; the size of sensitive data fragments is 64-128MB, and the number of fragments is ≥2; the size of ordinary data fragments is 128-256MB, and the number of fragments is ≥1; the fragment ID consists of a timestamp, a sensitivity level identifier, and a random string.

[0021] Furthermore, in step S3, core sensitive data is symmetrically encrypted using the AES-256-GCM algorithm, and the AES key is asymmetrically encrypted using the RSA-2048 algorithm; sensitive data is symmetrically encrypted using the AES-256-GCM algorithm; and ordinary data is encrypted using the SM4 algorithm.

[0022] Furthermore, in step S4, the storage node uses disk encryption technology to store the encrypted shards locally, and the backup shards and the primary shards are synchronized in real time via the TLS 1.3 protocol to ensure data consistency.

[0023] Furthermore, in step S5, the period for hierarchical dynamic verification is set according to the sensitivity level: the verification period for core sensitive data is 1 hour, the verification period for sensitive data is 6 hours, and the verification period for ordinary data is 24 hours; during periodic verification, the root hash value is calculated by constructing a Merkle tree and compared with the root hash value in the metadata server to achieve tamper location.

[0024] Furthermore, in step S6, the shard migration process uses TLS 1.3 protocol for encrypted transmission, the audit node verifies the shard integrity in real time, and after the migration is completed, the management node updates the shard mapping relationship in the metadata server.

[0025] Furthermore, in step S7, the audit record includes the operation time, operation type, operation node, operator, shard ID, sensitivity level, and operation result. Key audit records are written into the blockchain to ensure that they cannot be tampered with.

[0026] Furthermore, the method can be applied to scenarios requiring high-security storage, such as finance, healthcare, and government affairs, and is suitable for the storage and verification of petabyte-level massive data.

[0027] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0028] 1. This invention proposes a dynamic adaptive sharding strategy, which dynamically adjusts the sharding granularity, quantity, and mapping relationship based on data sensitivity level and node load, solving problems such as rigidity, data skew, and node overload in existing sharding strategies. It introduces a dual encryption mechanism, employing differentiated encryption strategies for data with different sensitivity levels, and combines this with the Shamir secret sharing algorithm to achieve distributed key management, avoiding single-point key leakage and overcoming the limitations of existing single encryption and centralized key management. Furthermore, it constructs a hierarchical dynamic verification system, combining real-time verification, periodic verification, and anomaly verification, and uses the Merkle tree verification algorithm to locate tampering, overcoming the shortcomings of existing verification mechanisms such as being single-faceted, having poor fault tolerance, and being unable to locate the tampering location.

[0029] 2. This invention combines deep learning-based sensitive data identification with dynamic sharding to achieve precise classification and targeted protection of sensitive data; it adopts an improved consistent hashing algorithm and introduces a dynamic adjustment mechanism for virtual nodes to balance load balancing and scalability; it integrates blockchain technology to ensure the immutability and traceability of audit records, meeting compliance requirements; and it provides secure protection for shard migration and recovery, ensuring that data is not leaked or lost during the migration process, thereby improving system availability.

[0030] 3. This invention can be directly applied to distributed storage clusters without requiring large-scale modifications to existing clusters, and has strong compatibility; it adopts differentiated sharding, encryption, and verification strategies for data with different sensitivity levels, balancing security and storage efficiency, and can adapt to the needs of different scenarios such as finance, healthcare, and government; the entire process is automated, requiring no manual intervention, reducing the workload of management personnel; it has functions such as automatic node failure recovery and dynamic load balancing, improving the stability and reliability of the system.

[0031] 4. This invention constructs a comprehensive security protection system through multiple security mechanisms such as sensitive data classification, double encryption, layered verification, distributed key management, and blockchain auditing, effectively preventing security risks such as data leakage, tampering, and loss; the unique design of sharded IDs and full lifecycle auditing enable data traceability and further enhance data security. Attached Figure Description

[0032] Figure 1 This is a flowchart of the distributed data sharding secure storage and verification method of the present invention;

[0033] Figure 2 This is a flowchart of the data preprocessing and sensitivity identification process of the present invention;

[0034] Figure 3 This is a flowchart of the dynamic adaptive fragmentation generation process of the present invention;

[0035] Figure 4 This is a flowchart of the fragmented dual encryption process of the present invention;

[0036] Figure 5 This is a flowchart of the layered dynamic verification process. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please see Figure 1-5 This embodiment presents a distributed data sharding security storage and verification method applied to a distributed storage cluster, which includes a management node, multiple storage nodes, and an audit node. The method includes the following steps:

[0039] Step S1: Data Preprocessing and Sensitivity Identification

[0040] The management node receives the data to be stored and preprocesses it, including data deduplication, format standardization, and redundancy compression. At the same time, the preprocessed data is classified into three levels of sensitivity using a sensitive data identification algorithm: ordinary data, sensitive data, and core sensitive data. Core sensitive data is key data involving privacy and confidentiality, sensitive data is regular sensitive information that needs to be encrypted, and ordinary data is public data that does not require special protection.

[0041] The sensitive data identification algorithm uses a deep learning model, combining data feature extraction and keyword matching to automatically identify sensitive fields in the data (such as ID card numbers, bank card numbers, medical records, etc.), and automatically classifies the sensitivity level according to the type and importance of the sensitive fields, without the need for manual intervention, thus improving identification efficiency and accuracy.

[0042] Step S2: Dynamic Adaptive Fragment Generation

[0043] The management node uses a dynamic adaptive sharding algorithm to generate different sharding strategies based on data sensitivity level, data size, storage node load, and network bandwidth. The specific process is as follows:

[0044] S2.1: The management node obtains the real-time status of all storage nodes in the distributed storage cluster, including the remaining storage space, CPU utilization, memory usage and network transmission rate of the nodes, and filters out available storage nodes (remaining storage space ≥ preset threshold, CPU utilization ≤ preset threshold).

[0045] S2.2: Determine the granularity and number of fragments based on the data sensitivity level: Core sensitive data uses fine-grained fragmentation (fragment size 16-64MB), with ≥3 fragments; sensitive data uses medium-grained fragmentation (fragment size 64-128MB), with ≥2 fragments; ordinary data uses coarse-grained fragmentation (fragment size 128-256MB), with ≥1 fragment.

[0046] S2.3: An improved consistent hashing algorithm is adopted, which dynamically adjusts the shard mapping relationship based on node load. Each generated shard is mapped to a different available storage node to ensure load balancing of each storage node (the difference in storage volume of each node's shards is ≤ a preset difference threshold). At the same time, a unique shard identifier (ID) is assigned to each shard of core sensitive data and sensitive data. The shard ID consists of a timestamp, a sensitivity level identifier, and a random string to ensure the uniqueness and traceability of the shards.

[0047] The improved consistent hashing algorithm introduces a dynamic adjustment mechanism for virtual nodes based on traditional consistent hashing. It increases or decreases the number of virtual nodes in real time according to the node load to avoid data skew and shorten the data migration amount during shard migration, thereby improving system scalability.

[0048] Step S3: Fragmented Double Encryption Processing

[0049] The management node performs double encryption on the shards generated in step S2, with different encryption strategies used for shards of different sensitivity levels, as detailed below:

[0050] S3.1: Core Sensitive Data Fragmentation: Employs dual encryption using "symmetric encryption + asymmetric encryption". First, the fragmented data is symmetrically encrypted using the AES-256-GCM algorithm to generate encrypted fragmented data. Then, the AES key is asymmetrically encrypted using the RSA-2048 algorithm to generate an encryption key. The encrypted fragmented data is then bound to the encryption key to form the core encrypted fragment.

[0051] S3.2: Sensitive data fragmentation: AES-256-GCM algorithm is used for symmetric encryption to generate sensitive encrypted fragments. The encryption key is split into multiple key fragments using the Shamir secret sharing algorithm and stored in different management nodes. The complete key can only be recovered by joint authorization of most key fragments to avoid single point of key leakage.

[0052] S3.3: Ordinary Data Sharding: Lightweight SM4 algorithm is used for encryption to generate ordinary encrypted shards, balancing security and storage efficiency;

[0053] After encryption is completed, the management node stores all encrypted fragments and their corresponding fragment IDs, sensitivity level identifiers, and encryption algorithm identifiers to a preset metadata server. The metadata server uses encrypted storage to prevent metadata leakage or tampering.

[0054] Step S4: Secure storage of encrypted fragments

[0055] The management node sends the encrypted fragments generated in step S3 to the corresponding storage nodes according to the fragment mapping relationship determined in step S2. After receiving the encrypted fragments, the storage nodes perform the following operations:

[0056] S4.1: The storage node performs a preliminary integrity check on the received encrypted fragment. It calculates the MD5 hash value of the encrypted fragment and compares it with the hash value sent by the management node. If they match, the encrypted fragment is accepted; if they do not match, the fragment is rejected and a retransmission request is sent to the management node.

[0057] S4.2: The storage node stores the received encrypted fragments in the local encrypted storage area. The local storage area uses disk encryption technology to prevent data leakage at the physical layer. At the same time, the storage node records the fragment storage time, storage location and receiving status, and synchronizes the recorded information to the management node and audit node.

[0058] S4.3: For core sensitive data shards, the management node controls the storage node to generate 2-3 backup shards for it. The backup shards are stored on storage nodes in different fault domains (such as different racks or different data centers) to ensure that data is not lost when a single node or fault domain fails. The backup shards are kept in real-time synchronization with the primary shards. The synchronization process uses the TLS1.3 protocol for encrypted transmission to prevent data leakage during transmission.

[0059] Step S5: Layered Dynamic Verification

[0060] Audit nodes and management nodes work together to perform layered dynamic verification of encrypted shards in storage nodes, including real-time verification, periodic verification, and anomaly verification, to ensure the integrity, consistency, and security of shard data. The specific process is as follows:

[0061] S5.1: Real-time verification: When a storage node receives an encrypted fragment, performs fragment backup, or fragment migration, the audit node obtains the operation data in real time, calculates the hash value (SHA-256) of the encrypted fragment, and compares it with the hash value stored in the metadata server. If the comparison is inconsistent, an abnormal alarm is immediately triggered and the abnormal information is recorded.

[0062] S5.2: Periodic Verification: Audit nodes set different verification cycles according to the data sensitivity level. The verification cycle for core sensitive data is 1 hour, the verification cycle for sensitive data is 6 hours, and the verification cycle for ordinary data is 24 hours. During periodic verification, the audit node uses the Merkle tree verification algorithm to construct a Merkle tree from the hash values ​​of all encrypted fragments in each storage node, calculate the root hash value, and compare it with the root hash value stored in the metadata server. If the comparison is inconsistent, the verification is traced layer by layer to locate the tampered or damaged fragment and record the verification result.

[0063] S5.3: Anomaly Verification: The management node monitors the operating status of the storage nodes in real time. When anomalies such as node failure, network interruption, or storage media damage are detected, the management node immediately notifies the audit node to perform a comprehensive verification of the encrypted fragments stored on that node. At the same time, when a user initiates a data read request, the audit node performs real-time verification of the read encrypted fragments to ensure that the read data has not been tampered with or damaged.

[0064] Step S6: Fragmented Migration and Recovery

[0065] When a storage node fails, is overloaded, or the distributed storage cluster is expanded or scaled down, the management node triggers a shard migration process, as follows:

[0066] S6.1: The management node selects the target storage node (that meets the available storage conditions and load balancing requirements) and sends migration instructions to the source storage node and the target storage node;

[0067] S6.2: The source storage node transmits the encrypted fragments to the target storage node via the TLS1.3 protocol. During the transmission, the audit node verifies the fragment integrity in real time. If data is lost or tampered with during the transmission, a retransmission is triggered immediately.

[0068] S6.3: After receiving the encrypted fragment, the target storage node completes the integrity verification and storage, and sends a migration completion notification to the management node; the management node updates the fragment mapping relationship, storage location and other information in the metadata server, and synchronizes it to the audit node;

[0069] When a storage node failure causes a fragment to be lost or damaged, the management node retrieves the corresponding backup fragment from the backup node based on the metadata information, restores it to the new storage node, and triggers the audit node to perform integrity verification to ensure the accuracy of data recovery.

[0070] Step S7: Full Lifecycle Audit and Traceability

[0071] The audit node records detailed operations throughout the entire lifecycle of data shards, including generation, encryption, storage, verification, migration, recovery, and deletion. The records include operation time, operation type, operation node, operator, shard ID, sensitivity level, and operation result. At the same time, the audit node uses blockchain technology to write key audit records into the blockchain, ensuring that the audit records are tamper-proof and traceable.

[0072] When data security issues occur (such as data leakage, tampering, or loss), managers can query the entire lifecycle of operation records through audit nodes to trace the root cause of the problem. At the same time, they can conduct responsibility determination and compliance audits based on the audit records in the blockchain. Example

[0073] This embodiment is applied to a distributed storage system for financial transaction data. The system includes one management node, ten storage nodes (divided into two fault domains, with five nodes in each domain), and one audit node. The data to be stored is financial transaction records (including core sensitive data: bank card number, transaction password; sensitive data: user name, contact information; and general data: transaction time, transaction type). The specific implementation steps are as follows:

[0074] Step S1: Data Preprocessing and Sensitivity Identification

[0075] The management node receives financial transaction record data, first deduplicating the data and deleting duplicate transaction records; then it standardizes the format, converting transaction data of different formats into JSON format; and uses the LZ4 compression algorithm to perform redundant compression on the standardized data to reduce storage overhead.

[0076] A CNN-based sensitive data identification model is used to extract field features from transaction data. Combined with keyword matching (such as "bank card number", "password", "name", etc.), it automatically identifies sensitive fields and divides the data into three levels: core sensitive data (bank card number, transaction password), sensitive data (user name, contact information), and ordinary data (transaction time, transaction type). The identification accuracy rate is ≥99.5%.

[0077] Step S2: Dynamic Adaptive Fragment Generation

[0078] S2.1: The management node obtains the status of 10 storage nodes in real time and filters out 8 available storage nodes with remaining storage space ≥100GB and CPU utilization ≤70%.

[0079] S2.2: Determine the fragmentation parameters based on the sensitivity level: Core sensitive data uses fine-grained fragmentation with a fragment size of 32MB and a fragment quantity of 3; sensitive data uses medium-grained fragmentation with a fragment size of 128MB and a fragment quantity of 2; ordinary data uses coarse-grained fragmentation with a fragment size of 256MB and a fragment quantity of 1.

[0080] S2.3: An improved consistent hashing algorithm is adopted to map each shard to different available storage nodes. Specifically, the three core sensitive data shards are mapped to storage nodes in three different fault domains, the two sensitive data shards are mapped to two different storage nodes, and the ordinary data shards are mapped to one storage node. At the same time, a unique shard ID is assigned to each shard, in the format of "timestamp_sensitivity level identifier_random string" (e.g., "20260325100000_01_abc123", where 01 represents core sensitive data, 02 represents sensitive data, and 03 represents ordinary data), to ensure the uniqueness of the shards.

[0081] Step S3: Fragmented Double Encryption Processing

[0082] S3.1: Core Sensitive Data Fragmentation: The fragmented data is symmetrically encrypted using the AES-256-GCM algorithm to generate encrypted fragmented data; the AES key is asymmetrically encrypted using the RSA-2048 algorithm to generate an encryption key; the encrypted fragmented data is bound to the encryption key to form the core encrypted fragment;

[0083] S3.2: Sensitive Data Fragmentation: AES-256-GCM algorithm is used for symmetric encryption to generate sensitive encrypted fragments; the AES key is split into 3 key fragments using the Shamir secret sharing algorithm, and stored in 3 different management nodes (backup management nodes). At least 2 key fragments need to be jointly authorized to recover the complete key;

[0084] S3.3: Ordinary data fragmentation: Encryption is performed using the SM4 algorithm to generate ordinary encrypted fragments;

[0085] After encryption is completed, the fragment ID, sensitivity level identifier, encryption algorithm identifier, and hash value (SHA-256) of all encrypted fragments are stored in the metadata server, which uses AES-256 encryption for storage.

[0086] Step S4: Secure storage of encrypted fragments

[0087] S4.1: The management node sends the encrypted fragment to the corresponding storage node. After receiving it, the storage node calculates the MD5 hash value of the encrypted fragment and compares it with the MD5 hash value sent by the management node. If they match, the storage node accepts the fragment; otherwise, it requests a retransmission.

[0088] S4.2: The storage node stores the encrypted fragments to the local encrypted disk (using BitLocker disk encryption technology), and records the fragment storage time, storage path and receiving status, and synchronizes them to the management node and audit node;

[0089] S4.3: Generate two backup shards for each of the three core sensitive data shards. The backup shards are stored on storage nodes in different fault domains (not in the same fault domain as the primary shard). The backup shards and the primary shards are synchronized in real time via the TLS 1.3 protocol to ensure data consistency.

[0090] Step S5: Layered Dynamic Verification

[0091] S5.1: Real-time verification: When a storage node receives an encrypted fragment, performs backup or migration, the audit node calculates the SHA-256 hash value of the encrypted fragment in real time and compares it with the hash value in the metadata server. If the comparison is inconsistent, an audible and visual alarm is triggered and abnormal information (such as abnormal node, abnormal fragment ID, and abnormal time) is recorded.

[0092] S5.2: Periodic Verification: Core sensitive data is verified every hour, sensitive data is verified every 6 hours, and ordinary data is verified every 24 hours. During verification, the audit node constructs a Merkle tree from the encrypted shard hash values ​​of each storage node, calculates the root hash value, and compares it with the root hash value in the metadata server. If they do not match, the tampered shard is located layer by layer. For example, if a core sensitive data shard is tampered with, the root hash value of the Merkle tree changes. The audit node can quickly locate the tampered shard by comparing the hash values ​​of the leaf nodes and non-leaf nodes layer by layer.

[0093] S5.3: Anomaly Verification: When the CPU utilization of a storage node reaches 90% (overload), the management node notifies the audit node to perform a full verification of the encrypted shards stored on that node. If no anomalies are found, shard migration is triggered. When a user reads transaction data, the audit node performs real-time verification of the read encrypted shards to ensure that the data has not been tampered with.

[0094] Step S6: Fragmented Migration and Recovery

[0095] When a storage node experiences a hardware failure (cannot operate normally), the management node selects target storage nodes with remaining storage space ≥100GB and CPU utilization ≤60%, and sends migration instructions to the source failure node (cannot operate, and retrieves fragments from the backup node) and the target storage node.

[0096] The backup node transmits the backup fragments of core sensitive data, the main fragments of sensitive data and ordinary data to the target storage node in encrypted form via the TLS 1.3 protocol. The audit node verifies the fragment integrity in real time during the transmission process to ensure that no data is lost or tampered with.

[0097] After the target storage node receives and stores the encrypted fragments, it sends a migration completion notification to the management node. The management node updates the fragment mapping relationship and storage location in the metadata server and synchronizes it to the audit node. After the faulty node is repaired, the management node uses it as a backup node, synchronizes the relevant fragment data, and restores normal operation.

[0098] Step S7: Full Lifecycle Audit and Traceability

[0099] The audit node records all operations: shard generation time, encryption time, storage location, verification results, migration records, recovery records, etc., and the recorded content is associated with the shard ID and the operating node; at the same time, key audit records (such as shard encryption, tampering anomalies, migration operations) are written to the blockchain to ensure that the audit records are immutable;

[0100] When there is suspicion of transaction data leakage, managers can query the full lifecycle records of the corresponding data shards through audit nodes, trace the operation process, locate the leakage node and the responsible person, and at the same time, ensure the authenticity and authority of the tracing results through blockchain audit records, so as to meet the compliance audit requirements of the financial field.

[0101] In this embodiment, the storage security of financial transaction data is significantly improved by the method of the present invention, the risk of sensitive data leakage is reduced by more than 99%, the sharding verification accuracy reaches 100%, the node failure recovery time is ≤5 minutes, and the system load balancing rate is ≥90%, which fully meets the storage and security needs of massive transaction data in the financial field.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A distributed data sharding secure storage and verification method, characterized in that, The method includes the following steps: Step S1: Data Preprocessing and Sensitivity Identification: The management node receives the data to be stored, performs data deduplication, format standardization, and redundancy compression; and classifies the preprocessed data into sensitivity levels, namely ordinary data, sensitive data, and core sensitive data, using a sensitive data identification algorithm. Step S2: Dynamic Adaptive Sharding Generation: The management node generates a dynamic sharding strategy based on the data sensitivity level, data size, storage node load, and network bandwidth using an improved consistent hashing algorithm. The data is sharded, each shard is assigned a unique shard ID, and the shards are mapped to different available storage nodes. Step S3: Fragmented Dual Encryption Processing: Differentiated encryption strategies are adopted according to the data sensitivity level. Core sensitive data is encrypted with "symmetric encryption + asymmetric encryption" dual encryption, sensitive data is encrypted with symmetric encryption and the key is distributed and stored through the Shamir secret sharing algorithm, and ordinary data is encrypted with lightweight symmetric encryption to generate encrypted fragments. Step S4: Secure storage of encrypted fragments: The management node sends the encrypted fragments to the corresponding storage nodes. After the storage nodes perform a preliminary integrity check, they store the fragments in the local encrypted storage area. At the same time, backup fragments are generated for the core sensitive data fragments and stored in storage nodes in different fault domains. Step S5: Layered Dynamic Verification: Audit nodes and management nodes collaborate to perform real-time verification, periodic verification, and anomaly verification on encrypted fragments. The Merkle tree verification algorithm is used to locate tampering and ensure data integrity and consistency. Step S6: Shard Migration and Recovery: When a storage node fails, is overloaded, or the cluster is expanded or scaled down, the management node triggers shard migration. The migration process uses encrypted transmission and real-time verification. When a shard is lost or damaged, data is restored from the backup node and verified. Step S7: Full lifecycle audit and traceability: Audit nodes record the full lifecycle operations of the shards, and use blockchain technology to store key audit records, so as to achieve traceability of operations and immutability of records.

2. The distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S1, the sensitive data identification algorithm uses a deep learning model, combining data feature extraction and keyword matching to automatically identify sensitive fields and classify sensitivity levels without manual intervention.

3. The distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S2, the improved consistent hashing algorithm introduces a dynamic adjustment mechanism for virtual nodes, which increases or decreases the number of virtual nodes in real time according to the storage node load to ensure that the load of each storage node is balanced and the difference in the shard storage of each node is ≤ a preset difference threshold.

4. The distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S2, the granularity and number of fragments are determined according to the sensitivity level: the size of core sensitive data fragments is 16-64MB, and the number of fragments is ≥3; the size of sensitive data fragments is 64-128MB, and the number of fragments is ≥2; the size of ordinary data fragments is 128-256MB, and the number of fragments is ≥1; the fragment ID consists of a timestamp, a sensitivity level identifier, and a random string.

5. A distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S3, core sensitive data is symmetrically encrypted using the AES-256-GCM algorithm, and the AES key is asymmetrically encrypted using the RSA-2048 algorithm; sensitive data is symmetrically encrypted using the AES-256-GCM algorithm; and ordinary data is encrypted using the SM4 algorithm.

6. The distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S4, the storage node uses disk encryption technology to store the encrypted fragments locally, and the backup fragments and the main fragments are synchronized in real time through the TLS1.3 protocol to ensure data consistency.

7. A distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S5, the period for hierarchical dynamic verification is set according to the sensitivity level: the verification period for core sensitive data is 1 hour, the verification period for sensitive data is 6 hours, and the verification period for ordinary data is 24 hours. During periodic verification, the root hash value is calculated by constructing a Merkle tree and compared with the root hash value in the metadata server to locate tampering.

8. A distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S6, the shard migration process uses TLS 1.3 protocol for encrypted transmission, the audit node verifies the shard integrity in real time, and after the migration is completed, the management node updates the shard mapping relationship in the metadata server.

9. A distributed data sharding secure storage and verification method according to claim 1, characterized in that, In step S7, the audit record includes operation time, operation type, operation node, operator, shard ID, sensitivity level, and operation result. Key audit records are written to the blockchain to ensure they cannot be tampered with.

10. The distributed data fragmentation secure storage and verification method according to any one of claims 1-9, characterized in that, The method can be applied to scenarios requiring high-security storage, such as finance, healthcare, and government affairs, and is suitable for the storage and verification of petabyte-level massive data.

Images