Method for safely transmitting and storing file fragments in distributed storage system

Abstract

The invention relates to the field of distributed storage, in particular to a method for safely transmitting and storing file fragments in a distributed storage system, and solves the problem that data are tampered or leaked in the transmission, storage and decryption process in the existing distributed storage field. The method comprises the following steps: step 1, data fragmentation; 2, data are uploaded before being uploaded; step 3, storing; 4, encrypting file link generation and forwarding; and 5, receiving and decrypting the file. According to the method, fragmented data, secret keys and parameters of each storage node in the distributed storage system are mastered by multiple parties, and encryption and decryption processing of organization, transmission and storage of the fragmented data is realized by adopting a stream encryption and parallel processing mode; the transmission and storage security of the file in the distributed system is effectively guaranteed, and the file is prevented from being tampered.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage, and in particular to a method for securely transmitting and storing file fragments in a distributed storage system. Background Art

[0002] Distributed storage systems are technologies that distribute data across multiple physical or virtual storage devices, typically located in different geographic locations or network nodes. The primary goal of this technology is to improve data reliability, availability, and access efficiency while reducing storage costs. Key features of distributed storage systems include data redundancy, horizontal scalability, load balancing, data sharding, fault tolerance and self-healing, location independence, scalability, cost-effectiveness, flexibility and diversity, and security. They are widely used in cloud computing platforms, big data platforms, multimedia services, and other fields.

[0003] While distributed storage systems are widely used, they also face various security risks and issues in data storage and transmission. The following is a detailed overview of the current major security risks and issues: Unauthorized access: Malicious users may gain access to sensitive data by defeating authentication mechanisms.

[0004] Man-in-the-middle attack: A network middleman may intercept and read the content of data transmission, resulting in the leakage of sensitive information.

[0005] Data tampering: During data transmission or storage, attackers can modify data without being detected, resulting in data integrity being compromised.

[0006] Key leakage: During the data encryption process, if the key is not managed properly, it may be leaked, resulting in data decryption.

[0007] Key distribution: In a distributed architecture, securely and efficiently distributing and managing encryption keys is a major challenge.

[0008] How senders can safely and securely transmit and store data in distributed storage systems, and how to safely, quickly, and conveniently share data stored in distributed systems with receivers are also technical directions and problems to be improved and optimized in current distributed storage systems.

[0009] There are many encryption methods in the prior art, including key derivation function, initialization vector, MAC algorithm, hash function with secret key: a hash algorithm with a randomly generated additional string as Salt; there is also a stream encryption method that combines encryption methods. Application No. 201911180760.4 discloses a data block confirmation method, device, equipment and medium based on blockchain, which adopts encryption methods such as process encryption and non-transmitter possession of passwords to encrypt, save and transmit data; however, its encryption requires the characteristics of the blockchain itself and cannot be applied to information encryption in other fields.

[0010] There is an urgent need for an encryption strategy that can effectively solve the problem of insufficient data storage and transmission security in the distributed storage field. Summary of the Invention

[0011] The present invention proposes a method for securely transmitting and storing file slices in a distributed storage system, which solves the problem of data being tampered with or leaked during transmission, storage and decryption in the existing distributed storage field.

[0012] The technical solution of the present invention is implemented as follows: a method for secure transmission and storage of file slices in a distributed storage system, comprising the following steps: Step 1: Data sharding: The uploader divides the file to be uploaded into slices of fixed size and numbers the slices; Step 2: Encryption before data upload: The uploader generates a random number as a master key; Step 3: Storage: The encrypted file slices are stored on the nodes of the distributed system and organized and managed by the distributed system; Step 4: Encryption of file link generation and forwarding: The distributed system node completes the storage of the file slice ciphertext, and the distributed system server generates a file link according to the file ID number and returns it to the uploader; The uploader performs simplified encoding of the file link and the master key in the form of Base64, etc., to synthesize an encrypted link and send it to the actual recipient of the file; Step 5: File reception and decryption: Step 501: The file receiver requests all distributed node addresses from the distributed storage system server based on the encrypted link, requests and obtains all file slice ciphertexts from the node, and the node returns the file slice ciphertext; Step 502: The receiver decrypts the slices in parallel.

[0013] A further technical solution is: the specific steps of step 2 are as follows: step 201: the uploader sends a file upload request to the distributed storage system server, and the system returns a SALT value, a task, and a key ID value parameter; step 202: the uploader generates a random number as a master key, and generates an IV value through a derivation algorithm and combines it with the SALT value returned in step 201 to generate a file encryption key E, and then sends the file upload request to the distributed storage system server again; step 203: the uploader combines the file shard number and IV value with the file key E generated in step 202 to generate a counter value for the file shard; step 204: the uploader uses the file key E and the counter value of each file shard to encrypt the plaintext of the file shard to obtain a shard ciphertext; step 205: the uploader account name, file size, and the hash value of the first and last two blocks are added as additional information to the file MAC value calculation; the file MAC value is combined with the shard MAC value of each file shard to obtain a final MAC value; the uploader combines the final MAC value with the simplified code to synthesize an encrypted link and sends it to the actual recipient of the file.

[0014] A further technical solution is: the keys, IV values, SALT values, shard lists and their node locations, and the nodes where the file shards are actually stored, used for encrypting the files and shards, are respectively controlled by the file uploader, the distributed storage system, the shard storage nodes, and the file receiver; wherein, the file protection key is generated, kept, and transmitted by the uploader; and the key ID sequence value and SALT value are generated, allocated, and maintained by the distributed system server.

[0015] The preferred technical solution is: the step 5 also includes 503 file verification: the recipient uses the user account, file size, and HASH values ​​of the first and last fragments to verify the file GMAC value. If the verification is successful, the file is correct and complete.

[0016] In a preferred technical solution, the uploader's sharded ciphertext generation in step 204 can be processed in parallel, the sharding is independent of the association between the upper and lower shards, and the shards can be of unequal length. The distributed storage node transmits and stores shards of varying sizes based on the node's network bandwidth and storage computing capabilities. Step 1 can perform a hash operation before sharding.

[0017] According to the preferred technical solution, the storage node storing the shard ciphertext can also perform a second encryption and storage on the shard ciphertext managed by the node according to the system's own security system software; when the user downloads, the storage node decrypts the shard ciphertext once, and the decryption is not the user-level encryption of the file shard. The present invention discloses an encryption method for storing and transmitting sharded data in a distributed storage system, which realizes the encryption and decryption processing of organizing, transmitting and storing sharded data by having the sharded data, keys and parameters of each storage node in the distributed storage system controlled by multiple parties, and adopting stream encryption and parallel processing mode; the present invention effectively guarantees the transmission and storage security of files in the distributed system, prevents files from being tampered with, and can perform parallel computing at the same time, which is not affected by the size of the shard, the number of copies and the order of the shards, thereby improving data processing performance; it is applicable to distributed storage systems deployed in local area networks and wide area networks, and effectively improves the data security of distributed storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 : The present invention is a flow chart of secure transmission and storage of file fragments in a distributed storage system. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention. Specific implementation method 1 A method for securely transmitting and storing file fragments in a distributed storage system comprises the following steps: Step 1: Data sharding: The uploader divides the file to be uploaded into shards of fixed size and numbers the shards. Step 2: Encryption before data upload: The uploader generates a random number as the master key; Step 201: The uploader sends a file upload request to the distributed storage system server, and the system returns the SALT value, task and key ID value parameters; Step 202: The uploader generates a random number as the master key, and generates an IV value through a derivation algorithm and combines it with the SALT value returned in step 201 to generate a file encryption key E, and then sends the file upload request to the distributed storage system server again; Step 203: The uploader combines the file shard number and IV value with the file key E generated in step 202 to generate a counter value for the file shard; Step 204: The uploader uses the file key E and the counter value of each file shard to encrypt the plaintext of the file shard to obtain the shard ciphertext; Step 205: The uploader's account name, file size, and the hash value of the first and last two blocks are added as additional information to the file MAC value calculation; the file MAC value is combined with the shard MAC value of each file shard to obtain the final MAC value.

[0022] Step 3: Storage: The encrypted file shards are stored on the nodes of the distributed system and organized and managed by the distributed system. The size, quantity, number of copies, and location of the shards are managed by the distributed system based on the performance of the nodes and the quality of the network. Step 4: File Link Generation and Forwarding Encryption: After the distributed system nodes complete the storage of the ciphertext file fragments, the distributed system server generates a file link based on the file ID number and returns it to the uploader. The file ID number contains the sequence number process, file number, and key used for this file transfer operation. This allows the distributed system server to locate the file transfer information, the storage node where the file fragment is located, and provides key index identification. In the security design of this embodiment, the distributed system maintains, stores, and manages the key IDs of user files. However, the actual keys for file transfer and storage are known only to the uploader and recipient. The file key ID is the unique identifier used by the distributed system server to maintain node information for the file fragment transfer and storage.

[0023] The uploader encodes the file link, master key, and file MAC value in a simplified form such as Base64 to form an encrypted link and sends it to the actual recipient of the file. Step 5: File reception and decryption: Step 501: The file receiver requests all distributed node addresses from the distributed storage system server based on the encrypted link, requests and obtains all file shard ciphertexts from the node, and the node returns the file shard ciphertext; Step 502: The receiver decrypts the shards in parallel based on the master key, SALT value, and shards.

[0024] During use, the uploader segments the file through the client, C / S software client or B / S software client, performs stream encryption, and encrypts and decrypts the transmitted and stored file segments, thereby implementing encryption and decryption processing for organizing, transmitting and storing the segmented data. This embodiment is a method for quickly and securely transmitting and storing files between a client and a distributed storage system, thereby improving the encryption level of files and enhancing data security. Specific embodiment 2 Based on the first embodiment, the key, IV value, salt value, shard list and its node location, and the actual storage node of the file shards used to encrypt files and shards are respectively controlled by the file uploader, the distributed storage system, the shard storage node, and the file receiver. A single party cannot obtain complete information related to the file. Even if some information is leaked or stolen, the security of the entire file can be guaranteed. The key for file protection is generated, stored, and transmitted by the uploader. The key ID sequence value and salt value are generated, distributed, and maintained by the distributed system server. This embodiment uses the multi-party storage and use of the password, with each party retaining only a portion. A single party does not have all the information and cannot decrypt the data, further improving the security and confidentiality of the file transmission and storage process. Specific embodiment three Based on the second specific embodiment, a hash operation can be performed before sharding to perform integrity backup. After receiving the file, a file verification is performed: the receiver uses the user account, file size, and the hash value of the first and last shards to verify the file's GMAC value. If the verification is successful, the file is correct and complete, further ensuring data security and integrity. Specific embodiment 4 On the basis of the above embodiment, the shard ciphertext generation can be processed in parallel, the sharding does not depend on the association between the upper and lower shards, nor is it affected by the shard size, the number of copies and the order of the shards, and the shards can be of different lengths. The distributed storage node transmits and stores the shards of different sizes according to the network bandwidth capacity and storage computing capacity of the node, so the file encryption and decryption process can be processed in parallel, reducing file processing and transmission time, improving file transmission efficiency, and providing convenient deployment conditions for heterogeneous nodes that are commonly found in distributed systems of wide area networks. Specific embodiment five Based on the above embodiment, the storage node storing the ciphertext of the shards can also perform a second encryption and storage of the ciphertext of the shards managed by the node based on the system's own security system software. When the user downloads the ciphertext of the shards, the storage node decrypts the ciphertext of the shards once. This first decryption is not the user-level encryption of the file shards. The storage node has no authority or ability to decrypt the user-level encryption of the file shards. This secondary encryption and decryption security scalability further enhances the security of file storage and transmission.

[0029] To achieve secure transmission and storage, you need to understand the following cryptographic technical knowledge (not limited to the following): Plaintext is the original information used as encryption input, that is, the original form of the message, usually denoted by m or p. The finite set of all possible plaintexts is called the plaintext space, usually denoted by M or P.

[0030] Ciphertext is the result of encryption transformation of plaintext, that is, the encrypted form of the message, usually represented by c.

[0031] The key is a parameter involved in cryptographic transformation, usually represented by k.

[0032] An encryption algorithm is a transformation function that transforms plaintext into ciphertext. The corresponding transformation process is called encryption, that is, the encoding process, usually represented by E, that is, c=Ek(P).

[0033] A decryption algorithm is a transformation function that converts ciphertext into plaintext. The corresponding transformation process is called decryption, or the decoding process, and is usually represented by D, i.e., p=Dk(c).

[0034] A key derivation function (KDF) generates one or more keys from a password. Specifically, it derives one or more keys from a master key, password, or passphrase using a PRF (Pseudo Random Function). This is a method for implementing key stretching (a slower hashing algorithm used to convert an initial key into a stronger key, intentionally increasing the time or space consumed during the calculation to protect weak passwords).

[0035] An Initialization Vector (IV) is a piece of data used in many cryptographic modes to randomize encryption. This allows different ciphertexts to be generated from the same plaintext and key, without the need for rekeying, a process that is often quite complex. Because the IV has different security requirements than the key, it generally does not need to be kept secret. However, in most cases, the same IV should not be used twice with the same key.

[0036] The key ID is public and identifies the key required to decrypt a file. A key is used when encrypting a file. A key is a piece of data used in an encryption algorithm to protect content. Each key is associated with a key ID. The key ID identifies the protected content from a licensing perspective. Although the same key ID can be used for multiple files, it is recommended to always use a unique key ID for each protected content. This encrypted transfer and storage method uses key ID strings to select the file and key to be transferred and stored. These strings contain the key ID encoded in base64.

[0037] MAC algorithm (Message Authentication Codes) Hash function with a secret key: The hash value of the message is controlled by a secret key K known only to the communicating parties. The hash value is called MAC.

[0038] To enhance the security of one-way hash calculations, salt is added to the hash algorithm. Salt acts as an encryption key, making it more difficult to crack. Commonly used one-way hash algorithms include MD5 and SHA.

[0039] In a hash algorithm, Salt is a randomly generated additional string that is used as input along with the original data to calculate the hash value. By adding Salt, the security and privacy of the hash value can be improved. Specifically, Salt can: Increase the probability of hash collisions, making it difficult for attackers to crack the original data; Prevent rainbow table attacks, because when using Salt, the attacker needs to calculate the hash value for each possible Salt; Increase the strength of passwords, because when using Salt, the hash value generated by the same password will be different.

[0040] Of course, without departing from the spirit and essence of the present invention, technicians familiar with the field should be able to make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for securely transmitting and storing file fragments in a distributed storage system, characterized by: The following steps are involved: Step 1: Data sharding: The uploader divides the file to be uploaded into shards of fixed size and numbers the shards. Step 2: Encryption before data upload: The uploader generates a random number as the master key; Step 3: Storage: The encrypted file fragments are stored on the nodes of the distributed system and organized and managed by the distributed system; Step 4: File link generation and forwarding encryption: The distributed system node completes the sharded ciphertext storage of the file, and the distributed system server generates a file link based on the file ID number and returns it to the uploader; The uploader encodes the file link and master key in a simplified format such as Base64 to create an encrypted link, which is then sent to the actual recipient of the file. Step 5: File reception and decryption: Step 501: The file recipient requests all distributed node addresses from the distributed storage system server based on the encrypted link, requests and obtains all file shard ciphertexts from the node, and the node returns the file shard ciphertexts; Step 502: The receiver decrypts the fragments in parallel.

2. The method for securely transmitting and storing file fragments in a distributed storage system according to claim 1, wherein: The specific steps of step 2 are: Step 201: The uploader sends a file upload request to the distributed storage system server, and the system returns the SALT value, task, and key ID value parameters; Step 202: The uploader generates a random number as the master key, and generates the file encryption key E by using the derivation algorithm and the generated IV value combined with the SALT value returned in step 201, and then sends the file upload request to the distributed storage system server again; Step 203: The uploader combines the file segment number and IV value with the file key E generated in step 202 to generate a counter value for the file segment; Step 204: The uploader uses the file key E and the counter value of each file segment to encrypt the plaintext of the file segment to obtain the segment ciphertext; Step 205: The uploader's account name, file size, and the hash values ​​of the first and last two blocks are added as additional information to the file MAC value calculation; the file MAC value is combined with the shard MAC value of each file shard to obtain the final MAC value; The uploader combines the final MAC value with the simplified encoding to form an encrypted link and sends it to the actual recipient of the file.

3. The method for securely transmitting and storing file fragments in a distributed storage system according to claim 3, wherein: The key, IV value, SALT value, shard list and its node location, and the actual storage node of the file shards used to encrypt the file and shards are respectively controlled by the file uploader, the distributed storage system, the shard storage node, and the file receiver; Among them, the file protection key is generated, kept and transmitted by the uploader; the key ID sequence value and SALT value are generated, distributed and maintained by the distributed system server.

4. A method for securely transmitting and storing file fragments in a distributed storage system according to any one of claims 1 to 3, characterized in that: The step 5 also includes 503 file verification: the recipient uses the user account, file size, and HASH values ​​of the first and last fragments to verify the file GMAC value. If the verification is successful, the file is correct and complete.

5. The method for securely transmitting and storing file fragments in a distributed storage system according to claim 4, characterized in that: In step 1, a hash operation may be performed before sharding.

6. The method for securely transmitting and storing file fragments in a distributed storage system according to claim 5, characterized in that: The generation of the fragmented ciphertext by the uploader in step 204 can be processed in parallel, the fragmentation does not depend on the association between the upper and lower fragments, and the fragments may not be of equal length. The distributed storage node transmits and stores the fragments of different sizes according to the network bandwidth capacity and storage computing capacity of the node.

7. The method for securely transmitting and storing file fragments in a distributed storage system according to claim 6, characterized in that: The storage node storing the shard ciphertext can also perform a second encryption and storage on the shard ciphertext managed by the node based on the system's own security system software; when the user downloads it, the storage node decrypts the shard ciphertext once, and the decryption is not a user-level encryption of the file shard.

Images