File storage method, device and server

By using the parallel processing characteristics of the HDFS distributed file system in Hadoop cluster, the target files are encrypted in pieces, solving the storage and reading security problems of large data volume and sensitive files in cloud storage, and achieving efficient and secure data processing and storage.

CN113032357BActive Publication Date: 2025-02-28INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202110477212.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-04-29
Publication Date
2025-02-28
Estimated Expiration
2041-04-29

AI Technical Summary

Technical Problem

The prior art is difficult to safely and efficiently store and read data files with a large amount of data in cloud storage, especially when it involves user privacy data, and cannot effectively protect data security.

Method used

By using the parallel processing characteristics of the HDFS distributed file system in Hadoop cluster, the target files are sliced ​​and encrypted, and the homomorphic encryption algorithm is used to encrypt the data to ensure the security of the data during storage and transmission.

Benefits of technology

It realizes efficient storage and reading of files with large amounts of data, ensures data security, improves user experience, and effectively protects user privacy data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

This specification provides a file storage method, device and server. After obtaining the target file to be stored, the node server in the Hadoop cluster first divides the plaintext data of the target file into multiple shard data in sequence according to the preset sharding rules; at the same time, it determines and sets corresponding association identifiers for each shard data according to the association parameters of each shard data about the target file; then calls multiple processing nodes in the HDFS distributed file system, uses the preset encryption rules, and encrypts the multiple shard data in parallel to obtain the ciphertext data of the multiple shard data; according to the association identifier, the ciphertext data of the multiple shard data are merged in sequence to obtain the ciphertext data of the target file, and stores it. In this way, the distributed characteristics of the HDFS distributed file system can be fully utilized, and target files with large data volumes can be safely and efficiently encrypted in a parallel manner before being stored in the cloud.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and more particularly to a file storage method, device, and server. Background Art

[0002] With the promotion and popularization of cloud storage services, more and more users are getting used to using cloud storage services to store data files, and users' requirements for cloud storage services are becoming higher and higher.

[0003] Specifically, in some application scenarios, the amount of data files that users need to store is often large, for example, it may reach more than a dozen TB, but users hope to be able to store and read the above data files more efficiently through the cloud; in addition, the data content of the above data files that need to be stored is usually more important and sensitive to users, and may also involve users' privacy data. Users do not want the above stored data files to be leaked, and have high requirements for the security of data files.

[0004] However, existing file storage methods often find it difficult to meet the needs of users in the above application scenarios, and are unable to safely and efficiently store and read data files with large amounts of data, which in turn affects the user experience.

[0005] To address the above problems, no effective solution has been proposed yet. Summary of the invention

[0006] This specification provides a file storage method, device and server, which can make full use of the distributed characteristics of the HDFS distributed file system based on the Hadoop cluster, and through parallel processing, safely and efficiently encrypt target files with large data volumes before storing them in the cloud, thereby improving the storage and reading efficiency of the target files, protecting the data security of the target files, and improving the user experience.

[0007] The embodiment of this specification provides a method for storing files, which is applied to a Hadoop cluster and includes:

[0008] Obtain the plaintext data of the target file to be stored;

[0009] According to the preset fragmentation rules, the plaintext data of the target file is sequentially divided into a plurality of fragmented data;

[0010] Determine and set a corresponding association identifier for the slice data according to an association parameter of the slice data with respect to the target file;

[0011] Calling multiple processing nodes in the HDFS distributed file system, using preset encryption rules, encrypting the multiple shard data in parallel to obtain ciphertext data of the multiple shard data;

[0012] According to the association identifier, the ciphertext data of the plurality of fragmented data are sequentially merged to obtain the ciphertext data of the target file;

[0013] The ciphertext data of the target file is stored.

[0014] In some embodiments, the data contained in the target file includes at least one of the following: transaction records of website users, account data of website users, and communication records of website users.

[0015] In some embodiments, according to a preset slicing rule, the plaintext data of the target file is sequentially divided into a plurality of slicing data, including:

[0016] According to the data characteristics of the target file, the plaintext data of the target file is divided into a plurality of fragmented data; wherein the data characteristics include at least one of the following: a source identifier of the data, a generation time of the data, and a type of the data.

[0017] In some embodiments, the preset encryption rules include encryption rules based on a homomorphic encryption algorithm.

[0018] In some embodiments, calling multiple processing nodes in the HDFS distributed file system, using preset encryption rules, encrypting the multiple shard data in parallel to obtain ciphertext data of the multiple shard data includes:

[0019] Generate a public key and a private key corresponding to the target file according to a preset encryption rule; and distribute the public key and the plurality of shard data to a plurality of processing nodes in the HDFS distributed file system;

[0020] The multiple processing nodes are called to encrypt the allocated shard data using the public key according to preset encryption rules to obtain ciphertext data of the multiple shard data.

[0021] In some embodiments, generating a public key and a private key corresponding to the target file according to a preset encryption rule includes:

[0022] According to a preset encryption rule, randomly generate a first odd prime number, a second odd prime number, a first integer, and a second integer;

[0023] generating an intermediate parameter according to the first odd prime number, the first integer and the second integer;

[0024] The public key is determined according to the intermediate parameter and the product of the first odd prime number and the second odd prime number; and the first odd prime number is determined as a private key corresponding to the public key.

[0025] In some embodiments, storing the ciphertext data of the target file includes:

[0026] Splitting the ciphertext data of the target file into multiple data components;

[0027] The multiple data components are respectively stored in multiple data nodes in the HDFS distributed file system, and storage information of the multiple data components is recorded.

[0028] In some embodiments, after storing the plurality of data components in a plurality of data nodes in the HDFS distributed file system respectively, the method further comprises:

[0029] Receiving a call request for a target file initiated by a terminal device;

[0030] In response to the call request, multiple data components of the ciphertext data of the target file and the splicing rules are sent to the terminal device; wherein the terminal device is used to obtain the ciphertext data of the target file by locally splicing the multiple data components according to the splicing rules.

[0031] In some embodiments, the call request also carries an identity identifier of a target user corresponding to the terminal device.

[0032] In some embodiments, the method further comprises:

[0033] Extracting the identity of the target user from the call request;

[0034] According to the identity identifier, verify the authority of the target user;

[0035] When the target user authority verification is passed, the private key corresponding to the target file is sent to the terminal device; wherein the terminal device is used to decrypt the ciphertext data of the target file locally using the private key to obtain the plaintext data of the target file.

[0036] The embodiment of the present specification also provides a file storage device, including:

[0037] An acquisition module, used to acquire the plaintext data of the target file to be stored;

[0038] A segmentation module, used to segment the plaintext data of the target file into a plurality of segment data in sequence according to a preset segmentation rule;

[0039] A determination module, used to determine and set a corresponding association identifier for the slice data according to an association parameter of the slice data with respect to the target file;

[0040] The calling module is used to call multiple processing nodes in the HDFS distributed file system, and encrypt the multiple shard data in parallel using preset encryption rules to obtain ciphertext data of the multiple shard data;

[0041] A merging module, used to merge the ciphertext data of the plurality of fragmented data in order according to the association identifier to obtain the ciphertext data of the target file;

[0042] A storage module is used to store the ciphertext data of the target file.

[0043] The embodiment of the present specification also provides a server, including a processor and a memory for storing processor executable instructions, wherein the processor implements the relevant steps of the file storage method when executing the instructions.

[0044] The embodiments of the present specification also provide a computer storage medium on which computer instructions are stored, and when the instructions are executed, the relevant steps of the file storage method are implemented.

[0045] This specification provides a file storage method, device and server. Based on this method, after obtaining the plaintext data of the target file to be stored, the node server in the Hadoop cluster can first divide the plaintext data of the target file into multiple shard data in sequence according to the preset sharding rules; at the same time, determine and set corresponding association identifiers for each shard data according to the association parameters of the shard data about the target file; then call multiple processing nodes in the HDFS distributed file system, use the preset encryption rules, and encrypt the multiple shard data in parallel to obtain the ciphertext data of the multiple shard data; then, according to the association identifier, the ciphertext data of the multiple shard data can be merged in sequence to obtain the ciphertext data of the target file, and then store it. In this way, the distributed characteristics of the HDFS distributed file system based on the Hadoop cluster can be fully utilized, and the target file with a large amount of data can be safely and efficiently encrypted and then stored in the cloud through parallel processing, which improves the storage and reading efficiency of the target file, protects the data security of the target file, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of this specification, the drawings required for use in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0047] Figure 1 It is a schematic diagram of an embodiment of the structure of a system using the file storage method provided by the embodiment of this specification;

[0048] Figure 2 is a flowchart of a method for storing files provided by an embodiment of this specification;

[0049] Figure 3 It is a schematic diagram of the structure of a server provided by an embodiment of this specification;

[0050] Figure 4 is a schematic diagram of the structure of a file storage device provided by an embodiment of the present specification;

[0051] Figure 5 It is a schematic diagram of an embodiment of a file storage method provided by an embodiment of this specification, in a scenario example;

[0052] Figure 6 It is a schematic diagram of an embodiment of a file storage method provided by an embodiment of this specification, in a scenario example;

[0053] Figure 7 It is a schematic diagram of an embodiment of a file storage method provided by an embodiment of this specification, in a scenario example. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0055] The present specification provides a method for storing files, which can be applied to a system including a Hadoop cluster and a terminal device. Figure 1 The Hadoop cluster may specifically include multiple node servers, and the terminal device may be connected to the node servers in the Hadoop cluster via wired or wireless means to perform specific data interaction.

[0056] In this embodiment, the Hadoop cluster can be specifically understood as a distributed framework based on Hadoop, which is a server cluster composed of multiple node servers as distributed processing nodes. The Hadoop cluster can be specifically used to provide cloud storage and cloud computing and other related services.

[0057] The above-mentioned Hadoop (full name Hadoop Distributed File System) can be specifically understood as a distributed framework based on a distributed computing and storage model, suitable for processing massive data. Hadoop has the characteristics of high fault tolerance and strong scalability.

[0058] Specifically, the above-mentioned Hadoop may include the following three parts: Hadoop Common, HDFS, and MapReduce. Among them, the above-mentioned Hadoop Common may specifically refer to a general tool set. The above-mentioned HDFS (Hadoop Distributed File System) may specifically refer to a distributed file system based on Hadoop with high fault tolerance, which is suitable for the storage of massive data. The above-mentioned Mapreduce may specifically refer to a distributed computing framework, which is suitable for the calculation of massive data.

[0059] In this embodiment, the node server constituting the Hadoop cluster may specifically include a background server capable of realizing functions such as data transmission and data processing. Specifically, the node server may be, for example, an electronic device having data calculation, storage and network interaction functions. Alternatively, the node server may also be a software program running in the electronic device to provide support for data processing, storage and network interaction. In this embodiment, the number of servers included in the node server is not specifically limited. The node server may specifically be one server, or several servers, or a server cluster formed by several servers.

[0060] In this embodiment, the terminal device may specifically include a front-end electronic device applied to the user side and capable of realizing functions such as data collection and data transmission. Specifically, the terminal device may be, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, etc. Alternatively, the terminal device may also be a software application that can be run in the above electronic devices. For example, it may be an APP running on a smart phone.

[0061] In specific implementation, when a user wants to store a target file, the user can use a terminal device to generate and send a storage request to a node server 1 in the Hadoop cluster that is connected to the terminal device.

[0062] The storage request may carry the plaintext data of the target file to be stored. The target file may be a file data with a large data volume and requiring privacy protection. For example, the target file may be a file data containing transaction records of website users and having a data volume of TB level.

[0063] Correspondingly, the node server 2 responsible for file storage processing in the Hadoop cluster (for example, the master node server in the cluster) receives the storage request through the node server 1 connected to the terminal device to obtain the plaintext data of the target file to be stored carried by the storage request.

[0064] Next, the node server 2 in the Hadoop cluster can first divide the plaintext data of the target file with a relatively large overall data volume into multiple shard data with relatively small data volumes in a certain order according to a preset sharding rule (for example, a sharding rule based on the transaction time of the transaction record).

[0065] While the plain text data of the target file is segmented in the above manner, the node server can also determine the associated parameters of the target file for each shard data (for example, the transaction time interval corresponding to each shard data) according to the preset sharding rules, and set an associated identifier corresponding to the associated parameter for each shard data. For example, for the shard data whose associated parameter is the first transaction time interval, the associated identifier can be set to "1"; for the shard data whose associated parameter is the second transaction time interval, the corresponding associated identifier can be set to "2", etc.

[0066] Furthermore, the node server 2 can call multiple processing nodes (i.e., multiple DataNodes) in the HDFS distributed file system in the Hadoop cluster, and use the preset encryption rules based on the homomorphic encryption algorithm to encrypt the multiple shard data in parallel, thereby effectively improving the overall encryption processing efficiency and quickly obtaining the corresponding ciphertext data of the multiple shard data.

[0067] When performing encryption, the node server 2 can first generate a key pair corresponding to the target file according to the preset encryption rules, including: a public key (which can be recorded as pk) and a private key (which can be recorded as sk). Then, the node server 2 can send the public key, the preset encryption rules, and multiple shard data to the corresponding processing nodes in the HDFS distributed file system respectively; then call multiple processing nodes in the HDFS distributed file system, and use the preset encryption rules and the received public key to encrypt the assigned shard data in parallel to obtain the ciphertext data of the shard data. At the same time, the node server 2 can provide the private key to the terminal device. The terminal device can save the private key locally on the terminal device in response to user instructions.

[0068] Then, the node server 2 can obtain the ciphertext data of multiple shard data encrypted by multiple node servers; then, according to the association identifiers carried by the ciphertext data of the shard data, the ciphertext data of the multiple shard data are merged in order to obtain the ciphertext data of the complete target file. For example, the node server 2 can merge in order from small to large the ciphertext data of the shard data with the association identifier "1", the ciphertext data of the shard data with the association identifier "2", the ciphertext data of the shard data with the association identifier "3", etc., to obtain the ciphertext data of the target file.

[0069] Through the above method, the parallel computing advantages of the distributed system framework can be fully utilized to efficiently complete the encryption processing for target files with large data volumes and obtain the ciphertext data of the target files.

[0070] After completing the encryption process of the target file and obtaining the ciphertext data of the target file in the above manner, the node server 2 can split the complete ciphertext data of the target file into multiple data components; and store the above multiple data components in multiple data nodes in the HDFS distributed file system respectively.

[0071] At the same time, the node server 2 will also determine and save information such as the node identifier of the data node stored in each data component in the multiple data components, the splicing rules between the data components, and the file identifier of the target file corresponding to each data component, as storage information of the data component.

[0072] This can effectively respond to storage requests and complete cloud storage for target files more safely and efficiently.

[0073] Through the above method, the distributed storage advantage of the distributed system framework can be fully utilized, and multiple data nodes in the HDFS distributed file system can be called to efficiently and securely complete the cloud storage of multiple data components of the ciphertext data of the target file.

[0074] When a user needs to download and use a target file, the user can use a terminal device to generate and send a call request for the target file to the node server 1 in the Hadoop cluster. The call request can at least carry a file identifier of the target file. The call request can further carry an identity identifier of the user.

[0075] Correspondingly, the node server 3 responsible for file download processing in the Hadoop cluster receives the call request through the node server 1 connected to the terminal device, and extracts the file identifier of the target file to be called and the user's identity identifier from the call request.

[0076] Next, the node server 3 may first perform authority verification on the user according to the user's identity identifier to determine whether the user who initiates the call request has the authority to call the target file.

[0077] If the user authority verification is passed, the node server 3 can find the storage information of multiple data components containing the file identifier of the target file based on the file identifier of the target file; and determine the node identifiers of the data nodes where the multiple data components are stored and the splicing rules between the data components based on the storage information.

[0078] Furthermore, the node server 3 may obtain the stored data components from the corresponding data node according to the node identifier of the data node; and send the obtained data components and the splicing rules between the data components to the terminal device.

[0079] Correspondingly, the terminal device receives and obtains multiple data components and the splicing rules between the multiple data components. In addition, the terminal device can respond to the user's instructions and first splice the multiple data components locally according to the splicing rules to obtain the complete ciphertext data of the target file; then use the locally stored private key to decrypt the ciphertext data of the target file to obtain the plaintext data of the target file. Furthermore, the terminal can display the plaintext data of the target file to the user for the user to use.

[0080] Through the above method, the characteristics of the distributed system framework can be fully utilized, so that users can conveniently and safely read and use target files stored in the cloud, thereby improving the user experience.

[0081] See also Figure 2 As shown, the embodiment of this specification provides a method for storing files. The method is specifically applied to a node server responsible for file storage processing in a Hadoop cluster. When implemented specifically, the method may include the following contents:

[0082] S201: Obtaining plaintext data of a target file to be stored;

[0083] S202: According to a preset fragmentation rule, the plaintext data of the target file is sequentially divided into a plurality of fragmented data;

[0084] S203: Determine and set a corresponding association identifier for the slice data according to the association parameter of the slice data with respect to the target file;

[0085] S204: calling multiple processing nodes in the HDFS distributed file system, and using preset encryption rules to perform encryption processing on the multiple shard data in parallel to obtain ciphertext data of the multiple shard data;

[0086] S205: According to the association identifier, merging the ciphertext data of the plurality of fragmented data in order to obtain the ciphertext data of the target file;

[0087] S206: Store the ciphertext data of the target file.

[0088] Through the above embodiments, we can make full use of the parallel computing, distributed storage and other characteristics of the HDFS distributed file system based on the Hadoop cluster, adopt parallel processing to safely and efficiently encrypt target files with large data volumes and then perform corresponding cloud storage, thereby improving the storage and reading efficiency of the target files, protecting the data security of the target files, and improving the user experience.

[0089] In some embodiments, the target file may specifically be a data file with a relatively large data volume and requiring data privacy protection.

[0090] In some embodiments, the data contained in the target file may specifically include at least one of the following: transaction records of website users, account data of website users, communication records of website users, etc.

[0091] Of course, it should be noted that the data contained in the target files listed above are only a schematic illustration. In specific implementation, according to specific application scenarios and processing requirements, the data contained in the target files may also be other types of business data. This specification does not limit this.

[0092] Through the above embodiments, the file storage method provided in this specification can be generalized and applied to various application scenarios, so as to efficiently and securely perform corresponding storage processing on target files containing different types of business data.

[0093] In some embodiments, during specific implementation, the target user may use a terminal device to generate a storage request for indicating the storage of the target file. The storage request may carry the target file to be stored. The terminal device may send the storage request to a node server in the Hadoop cluster. Accordingly, the Hadoop cluster may receive the storage request through the node server to obtain the plaintext data of the target file to be stored.

[0094] In some embodiments, the above-mentioned method of dividing the plaintext data of the target file into multiple fragmented data in sequence according to a preset fragmentation rule may include the following contents during specific implementation: dividing the plaintext data of the target file into multiple fragmented data according to data characteristics of the target file; wherein the data characteristics include at least one of the following: source identification of the data, generation time of the data, and type of the data.

[0095] Through the above embodiments, according to the preset fragmentation rules, the plaintext data of the target file with a relatively large data volume can be divided into multiple fragmented data with relatively small data volumes in a certain order to facilitate subsequent processing.

[0096] In some embodiments, the preset sharding rule can be specifically understood as a rule data that records the data features relied upon for sharding and the specific sharding method. The data features can specifically include: data source identification, data generation time, data type, etc. The preset sharding rule matches the data contained in the target file.

[0097] Through the above embodiments, the data contained in the target file can be divided into a plurality of corresponding fragmented data more accurately and reasonably according to the preset fragmentation rules.

[0098] In some embodiments, before implementation, corresponding preset sharding rules may be configured for a variety of different application scenarios based on the data characteristics of the target files in different application scenarios. The data characteristics and sharding methods recorded in the preset sharding rules corresponding to different application scenarios may be different.

[0099] Specifically, for example, corresponding to the transaction data processing scenario, in the configured preset sharding rules, the data features may include the source identifier of the data, such as the domain name of the shopping website where the transaction record occurred; the sharding method may include dividing the transaction records with the same source identifier of the data into one shard data.

[0100] For another example, corresponding to the user's communication record management scenario, in the configured preset sharding rules, data features may include: data generation time, for example, the generation date of the communication record; sharding methods may include dividing communication records with the same data generation time into one shard data, etc.

[0101] In some embodiments, during specific implementation, the target application scenario corresponding to the target file may be determined first; then, according to the multiple preset slicing rules of the application scenario, a preset slicing rule corresponding to the target application scenario may be determined as a preset slicing rule that matches the data contained in the target file. Then, according to the preset slicing rule, the target file may be specifically segmented.

[0102] In some embodiments, the above-mentioned association parameters can be specifically used to characterize the association relationship between the shard data segmented based on the preset sharding rule and the target file.

[0103] In the process of sequentially dividing the plaintext data of the target file into multiple shard data according to the preset sharding rules, the associated parameters of the shard data with respect to the target file can be determined according to the data characteristics of a shard data and the sharding order of the shard data.

[0104] Specifically, for example, when the communication record with a generation date of April 26, 2021 is cut out from the plaintext data of the target file according to the preset fragmentation rule as the third fragmentation data, the associated parameter of the fragmentation data can be determined as: 2021.04.26-3. That is, it means that the data feature corresponding to the fragmentation data is the date April 26, 2021, and it belongs to the third fragmentation data in the order of the target file.

[0105] After determining the association parameters of the shard data, corresponding association identifiers can be set for each shard data according to the association parameters of the shard data. Specifically, for example, an association identifier "3" can be set for the third shard data.

[0106] In some embodiments, according to the preset slicing rules, the data contained in the target file can also be sequentially segmented according to the preset slicing size (for example, the amount of data contained in each slicing), so that the amount of data contained in each of the multiple slicing data obtained by segmentation can be equal or similar.

[0107] Furthermore, during the specific segmentation, each time a slice data of a preset slice size is segmented, it can also be marked in the target file at the same time. Specifically, for example, when the current slice data is segmented, the character "cur" can be used to mark the last data of the current slice data in the target file. Accordingly, when the next slice data after the current slice data is segmented, the data marked with the character "cur" can be first retrieved in the target file; then, starting from the next data after the data, the next slice data can be segmented. This can effectively avoid omissions or duplications during segmentation, improve the accuracy of the segmentation process, and reduce errors introduced during segmentation.

[0108] In some embodiments, the preset encryption rules may specifically include encryption rules based on a homomorphic encryption algorithm. The homomorphic encryption algorithm may be specifically understood as a cryptographic technique based on computational complexity theory of mathematical problems, and has the following characteristics: processing homomorphically encrypted data to obtain an output, and then decrypting the output, the result is the same as the output obtained by performing the same processing on the unencrypted original data.

[0109] Through the above embodiments, the preset encryption rules based on the homomorphic encryption algorithm can be used to perform encryption processing on the shard data of multiple target files in parallel with high reliability.

[0110] In some embodiments, the above-mentioned calling of multiple processing nodes in the HDFS distributed file system, using preset encryption rules, encrypts the multiple shard data in parallel to obtain ciphertext data of the multiple shard data. When implemented specifically, it may include the following contents: generating a public key and a private key corresponding to the target file according to the preset encryption rules; and assigning the public key and the multiple shard data to multiple processing nodes in the HDFS distributed file system; calling the multiple processing nodes, and respectively encrypting the assigned shard data using the public key according to the preset encryption rules to obtain the ciphertext data of the multiple shard data.

[0111] Through the above embodiments, the advantages of parallel computing of the HDFS distributed file system can be fully utilized to efficiently and securely complete the encryption processing of multiple shard data in a parallel manner.

[0112] In some embodiments, the above-mentioned generation of a public key and a private key corresponding to the target file according to a preset encryption rule may include the following contents during specific implementation:

[0113] S1: randomly generate a first odd prime number, a second odd prime number, a first integer, and a second integer according to a preset encryption rule;

[0114] S2: Generate an intermediate parameter according to the first odd prime number, the first integer and the second integer;

[0115] S3: Determine the public key according to the intermediate parameter and the product of the first odd prime number and the second odd prime number; and determine the first odd prime number as a private key corresponding to the public key.

[0116] Through the above embodiments, a public key and a private key with higher security can be generated according to the preset encryption rules.

[0117] In some embodiments, when the node server in the Hadoop cluster is implemented, the corresponding public key (pk) and private key (sk) can be generated according to the preset encryption rules and the corresponding key generation algorithm KeyGen(). Specifically, an odd prime number p of η' bits can be randomly selected as the first odd prime number, and an odd prime number q of θ bits can be randomly selected as the second odd prime number; and the product N = pq of the first odd prime number and the second odd prime number can be calculated. Then two integers l and h can be randomly selected in the following manner: l∈[0,2 γ / p],h∈(-2 ρ ,2 ρ ), respectively denoted as the first integer and the second integer; and according to the first integer and the second integer, the corresponding intermediate parameter is calculated according to the following formula: x=pl+2 k h. Further, the corresponding public key can be determined based on the above intermediate parameter, the product of the first odd prime number and the second odd prime number: pk = (N, x); at the same time, the corresponding private key can be determined based on the first odd prime number: sk = p.

[0118] According to the above method, the node server can calculate the corresponding public key and private key; then the shard data, together with the public key and the preset encryption rules, can be sent to multiple processing nodes (for example, DataNode nodes) in the HDFS distributed file system; then multiple processing nodes are called to encrypt the assigned shard data using the above public key in a parallel processing manner according to the preset encryption rules to obtain the ciphertext data of the multiple shard data.

[0119] Specifically, when calling the current processing node among multiple processing nodes to encrypt the assigned shard data, the processing node can be called to perform specific encryption processing on the assigned shard data m based on the following formula using the public key pk according to the preset encryption rules: Encrypt(pk, m). When using Encrypt(pk, m) to encrypt m, the processing node can first randomly select two random integers r1 and r2: r1∈(-2ρ', 2ρ'), r2∈(-2 ρ , 2 ρ); then, according to the public key pk = (N, x), calculate the ciphertext data of m according to the following formula: c = m + 2 k r1+r2 x mod N.

[0120] According to the above method, the node server can call multiple processing nodes in parallel to utilize preset encryption rules and public keys to efficiently encrypt the allocated component data to obtain the corresponding ciphertext data of the multiple component data.

[0121] Then, the node server can receive the ciphertext data of multiple component data after the encryption processing by the processing node. Further, the node server can sequentially merge the ciphertext data of the above multiple fragmented data according to the association identifier set in the ciphertext data of each component data, so as to obtain the ciphertext data of the complete target file. In this way, data encryption for target files with large data volume can be completed more efficiently.

[0122] In some embodiments, the above-mentioned storage of the ciphertext data of the target file may include the following contents when implemented specifically: splitting the ciphertext data of the target file into multiple data components; storing the multiple data components in multiple data nodes in the HDFS distributed file system respectively, and recording the storage information of the multiple data components.

[0123] Through the above embodiments, the distributed storage advantage of the HDFS distributed file system can be fully utilized to split the ciphertext data of a target file with a large amount of data into multiple data components for distributed cloud storage, thereby effectively improving the storage efficiency of the target file, and also facilitating subsequent users or other clusters to read and call the target file.

[0124] In some embodiments, during the specific splitting, the ciphertext data of the target file can be flexibly split into multiple data components according to the data volume of the ciphertext data of the target file and the remaining storage space of the available data nodes in the current HDFS distributed file system, so as to fully and effectively utilize storage resources and avoid wasting storage resources.

[0125] In some embodiments, when the node server stores multiple data components in multiple data nodes in the HDFS distributed file system respectively, it will also simultaneously determine and save information such as the node identifier of the data node where each data component in the multiple data components is stored, the splicing rules between the data components, and the file identifier of the target file corresponding to each data component, as storage information of the data component.

[0126] In some embodiments, after the multiple data components are respectively stored in multiple data nodes in the HDFS distributed file system, the method may further include the following when implemented:

[0127] S1: receiving a call request for a target file initiated by a terminal device;

[0128] S2: In response to the call request, multiple data components of the ciphertext data of the target file and the splicing rules are sent to the terminal device; wherein the terminal device is used to obtain the ciphertext data of the target file by locally splicing the multiple data components according to the splicing rules.

[0129] Through the above embodiments, when the target user needs to retrieve and use the target file stored in the cloud, a call request for the target file can be initiated through the terminal device to obtain multiple encrypted data components corresponding to the target file and corresponding splicing rules; then the target user can use the local computing resources of the terminal device to splice multiple data components according to the splicing rules locally on the terminal device to obtain the complete ciphertext data of the target file, thereby avoiding the occupation of the computing resources of the Hadoop cluster, so that the target user can download the target file efficiently and conveniently.

[0130] In some embodiments, the call request may also carry an identity identifier of a target user corresponding to the terminal device. The identity identifier may be a type of identification information that can indicate the identity of the target user. Specifically, the identity identifier may be the name of the target user, the ID number of the target user, the account name of the target user, the mobile phone number used by the target user for registration, etc.

[0131] Through the above embodiment, the target user can provide the node server with an identity identifier for authority verification through the initiated call request, thereby simplifying the user operation.

[0132] In some embodiments, the method may further include the following when implemented:

[0133] S1: extracting the identity of the target user from the call request;

[0134] S2: According to the identity identifier, verify the authority of the target user;

[0135] S3: When the target user authority verification is passed, the private key corresponding to the target file is sent to the terminal device; wherein the terminal device is used to decrypt the ciphertext data of the target file locally using the private key to obtain the plaintext data of the target file.

[0136] Through the above embodiment, the node server will only provide the corresponding private key to the terminal device if it verifies the target user's authority according to the target user's identity and determines that the target user has the authority to call the target file; the terminal device can then use the above private key to decrypt the ciphertext data of the target file locally to obtain the plaintext data of the target file and provide it to the target user. This can effectively prevent the node server from leaking the real content of the target file to users without authority, and protect the data privacy of the target file.

[0137] In some embodiments, when the terminal device uses the private key to decrypt the ciphertext data of the target file, it can specifically use the private key sk to decrypt the ciphertext data c of the target file according to the following formula Decrypt(sk, c) to obtain the plaintext data of the target file.

[0138] As can be seen from the above, the file storage method provided by the embodiment of this specification is based on this method. After the Hadoop cluster obtains the plaintext data of the target file to be stored, it can first divide the plaintext data of the target file into multiple shard data in sequence according to the preset sharding rules; at the same time, determine and set corresponding association identifiers for each shard data according to the association parameters of the shard data about the target file; then call multiple processing nodes in the HDFS distributed file system, use the preset encryption rules, and encrypt the multiple shard data in parallel to obtain the ciphertext data of the multiple shard data; then merge the ciphertext data of the multiple shard data in sequence according to the association identifier, obtain the ciphertext data of the target file, and then store it. In this way, the distributed characteristics of the HDFS distributed file system based on the Hadoop cluster can be fully utilized, and the target file with a large amount of data can be safely and efficiently encrypted through parallel processing before cloud storage, which improves the storage and reading efficiency of the target file, protects the data security of the target file, and improves the user experience.

[0139] The embodiment of the present specification also provides a server, including a processor and a memory for storing instructions executable by the processor, wherein the processor can perform the following steps according to the instructions when implemented: obtaining plaintext data of a target file to be stored;

[0140] According to the preset sharding rules, the plaintext data of the target file is divided into multiple shard data in sequence; the association parameters of the shard data with respect to the target file are determined and corresponding association identifiers are set for the shard data; multiple processing nodes in the HDFS distributed file system are called, and the multiple shard data are encrypted in parallel using the preset encryption rules to obtain ciphertext data of the multiple shard data; according to the association identifiers, the ciphertext data of the multiple shard data are merged in sequence to obtain the ciphertext data of the target file; and the ciphertext data of the target file is stored.

[0141] In order to complete the above instructions more accurately, refer to Figure 3 As shown, the embodiment of this specification also provides another specific server, wherein the server includes a network communication port 301, a processor 302 and a memory 303, and the above structures are connected through internal cables so that each structure can perform specific data interaction.

[0142] The network communication port 301 may be used to obtain plain text data of a target file to be stored.

[0143] The processor 302 can be specifically used to divide the plaintext data of the target file into multiple shard data in sequence according to a preset sharding rule; determine and set corresponding association identifiers for the shard data according to association parameters of the shard data with respect to the target file; call multiple processing nodes in the HDFS distributed file system, and use preset encryption rules to encrypt the multiple shard data in parallel to obtain ciphertext data of the multiple shard data; merge the ciphertext data of the multiple shard data in sequence according to the association identifier to obtain the ciphertext data of the target file; and store the ciphertext data of the target file.

[0144] The memory 303 may be specifically used to store corresponding instruction programs.

[0145] In this embodiment, the network communication port 301 can be a virtual port that is bound to different communication protocols so that different data can be sent or received. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. In addition, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM, CDMA, etc.; it can also be a Wifi chip; it can also be a Bluetooth chip.

[0146] In this embodiment, the processor 302 may be implemented in any appropriate manner. For example, the processor may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) executable by the (micro)processor, a logic gate, a switch, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, etc. This specification does not limit this.

[0147] In this embodiment, the memory 303 may include multiple levels. In a digital system, anything that can store binary data can be a memory; in an integrated circuit, a circuit with a storage function but no physical form is also called a memory, such as RAM, FIFO, etc.; in a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0148] The embodiment of the present specification also provides a computer storage medium based on the above-mentioned file storage method, wherein the computer storage medium stores computer program instructions, which, when executed, implement the following: obtaining plaintext data of a target file to be stored; dividing the plaintext data of the target file into a plurality of shard data in sequence according to a preset sharding rule; determining and setting a corresponding association identifier for the shard data according to association parameters of the shard data with respect to the target file; calling a plurality of processing nodes in an HDFS distributed file system, and using preset encryption rules, encrypting the plurality of shard data in parallel to obtain ciphertext data of the plurality of shard data; merging the ciphertext data of the plurality of shard data in sequence according to the association identifier to obtain the ciphertext data of the target file; and storing the ciphertext data of the target file.

[0149] In this embodiment, the storage medium includes, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a cache, a hard disk (HDD), or a memory card. The memory may be used to store computer program instructions. The network communication unit may be an interface for network connection communication set in accordance with the standard specified by the communication protocol.

[0150] In this embodiment, the functions and effects specifically implemented by the program instructions stored in the computer storage medium can be explained in comparison with other implementations and will not be described in detail here.

[0151] See also Figure 4As shown, at the software level, the embodiment of this specification also provides a file storage device, which may specifically include the following structural modules:

[0152] The acquisition module 401 may be specifically used to acquire the plaintext data of the target file to be stored;

[0153] The segmentation module 402 may be specifically configured to segment the plaintext data of the target file into a plurality of segment data in sequence according to a preset segmentation rule;

[0154] The determination module 403 may be specifically used to determine and set a corresponding association identifier for the segment data according to an association parameter of the segment data with respect to the target file;

[0155] The calling module 404 may be specifically used to call multiple processing nodes in the HDFS distributed file system, and encrypt the multiple shard data in parallel using a preset encryption rule to obtain ciphertext data of the multiple shard data;

[0156] The merging module 405 may be specifically configured to merge the ciphertext data of the plurality of fragmented data in order according to the association identifier to obtain the ciphertext data of the target file;

[0157] The storage module 406 may be specifically configured to store the ciphertext data of the target file.

[0158] It should be noted that the units, devices or modules described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. For the convenience of description, the above devices are described separately by functions divided into various modules. Of course, when implementing this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the modules that implement the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0159] As can be seen from the above, the file storage device provided in the embodiments of this specification can make full use of the distributed characteristics of the HDFS distributed file system based on the Hadoop cluster, and through parallel processing, safely and efficiently encrypt target files with large data volumes before storing them in the cloud, thereby improving the storage and reading efficiency of the target files, protecting the data security of the target files, and improving the user experience.

[0160] In a specific scenario example, the file storage method provided in this specification can be applied to perform distributed data encryption and cloud storage on big data files based on Hadoop.

[0161] In this scenario example, the method provided in this specification can be used to quickly encrypt a large data file (for example, a target file) into an unreadable and unattackable ciphertext (for example, the ciphertext data of the target file), thereby achieving fast encryption and decryption. In this way, even if an attacker obtains the intermediate data, the attacker cannot obtain the plaintext. In addition, when encrypting, homomorphic encryption is used (for example, based on preset encryption rules of a homomorphic encryption algorithm) to improve data security and solve the problem of data security in a distributed system. Moreover, during the calculation process, data transmission between multiple clusters can be supported, which solves the problem of data security, prevents attackers from obtaining sensitive data, and avoids data leakage.

[0162] For details, please refer to Figure 5 As shown, a distributed data encryption device based on Hadoop is constructed based on the file storage method provided in this specification. The device can specifically include: a data input module 1, a data file upload module 2, a data sharding algorithm 3, a data file parallel encryption processing module 4, an encrypted HDFS file DataNode storage module 5, a decryption module 6, and a data file plaintext data acquisition module 7.

[0163] Among them, the data input module 1 is connected to the data file upload module 2, the data file upload module 2 is connected to the data sharding algorithm 3, the data sharding algorithm 3 is connected to the data file parallel encryption processing module 4, the data file parallel encryption processing module 4 is connected to the encrypted HDFS file DataNode storage module 5, the encrypted HDFS file DataNode storage module 5 is connected to the decryption module 6, and the decryption module 6 is connected to the plaintext data module 7 for obtaining the data file.

[0164] Specifically, the data input module 1 is responsible for inputting the specific information of the large data set, such as the shopping records of a user in a quarter, including account passwords and order data (as the target file to be stored). It should be noted that these data involve user privacy and are large in TB level.

[0165] Based on the data file uploading module 2, after receiving the data to be encrypted, the Hadoop server cluster uploads the data files of these large data sets to the HDFS distributed file system of the Hadoop platform server.

[0166] Data sharding algorithm 3, the sharding algorithm can be executed according to the following procedure: the original file is divided into n shards (for example, multiple shard data), where L represents the size of the file, i represents the i-th shard, and li represents the size of the i-shard file. When sharding, cur can be used to represent the position of the data being processed in the original file, which is used to mark whether the file has been read.

[0167] The encrypted HDFS file DataNode storage module 5 and the data file parallel encryption processing module 4 can cooperate to perform specific encryption processing. For the specific encryption processing process, please refer to Figure 6 As shown, multiple plaintext fragments are encrypted in parallel. Specifically, it may include the following: After the plaintext is divided into n plaintexts of the same size according to the size of the cluster, the n plaintexts are encrypted in parallel using the homomorphic encryption algorithm. Specifically, the first step is to use the key generation algorithm KeyGen(): randomly select an odd prime number p of η' bits and an odd prime number q of θ bits, and let N = pq. Then select two random integers l∈[0, 2 γ / p],h∈(-2 ρ ,2 ρ ), and calculate x=pl+2 k h, we can get the public key pk = (N, x) and the private key sk = p. The second step is to use the encryption algorithm Encrypt() to encrypt the given message m∈{0,1}k. We can choose two random integers r1∈(-2ρ', 2ρ'), r2∈(-2 ρ , 2 ρ ), according to the public key pk = (N, x) we can calculate the ciphertext c = m + 2 k r1+r2 x mod N. After encryption, the ciphertexts are merged in order to obtain the encrypted data of the entire large data set (for example, the ciphertext data of the target file). Based on the above modules, the characteristics of the Hadoop system can be used to store the ciphertext data on different DataNode nodes.

[0168] Decryption processing module 6, for specific operation, please refer to Figure 7As shown. If the user needs data, or other clusters need data, the data can be taken out from the corresponding DataNode node and then decrypted. This is the reverse process of module 4. In this process, the decryption function of the homomorphic encryption algorithm can be used to decrypt the data set. Specifically, the decryption algorithm Decrypt(sk, c) can be used. Based on the given ciphertext data c, the corresponding plaintext can be decrypted using the private key: m = (c mod p) mod 2 k . Thus, the plain text of the entire data can be obtained.

[0169] Based on the plaintext data module 7 for obtaining data files, the requesting cluster or user can obtain the desired plaintext data.

[0170] Through the above scenario examples, based on the method ideas provided in this manual, the big data can be segmented using a block algorithm, and then each data block can be encrypted using a homomorphic algorithm using the parallel characteristics of the Hadoop cluster. Finally, these ciphertext blocks are merged to obtain the ciphertext of the entire data. In this way, the Hadoop-based distributed data encryption method can be used to help users store data more securely and efficiently, solving the data security problem in the big data cloud environment, and also solving the problem that data privacy in distributed systems is easily leaked, so that the security of large data sets is guaranteed.

[0171] Although the present specification provides method operation steps as described in the embodiments or flow charts, more or less operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent a unique execution order. When the device or client product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. The first, second, etc. words are used to represent the name, and do not represent any particular order.

[0172] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0173] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media including storage devices.

[0174] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that the present specification can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present specification can essentially be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a mobile terminal, a server, or a network device, etc.) to execute the methods described in each embodiment of the present specification or some parts of the embodiments.

[0175] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. This specification can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0176] Although the present specification is described through embodiments, those skilled in the art will appreciate that there are many modifications and changes to the present specification without departing from the spirit of the present specification, and it is intended that the appended claims include these modifications and changes without departing from the spirit of the present specification.

Claims

1. A method for storing a file, characterized in that: The method is applied to a Hadoop cluster and includes: Obtaining plaintext data of a target file to be stored; wherein the target file is a data file with a large amount of data and which requires data privacy protection; According to the preset fragmentation rules, the plaintext data of the target file is sequentially divided into a plurality of fragmented data; Determine and set a corresponding association identifier for the slice data according to an association parameter of the slice data with respect to the target file; wherein the association parameter is used to characterize the association relationship between the slice data divided based on the preset slice rule and the target file; Calling multiple processing nodes in the HDFS distributed file system, using preset encryption rules, encrypting the multiple shard data in parallel to obtain ciphertext data of the multiple shard data; According to the association identifier, the ciphertext data of the plurality of fragmented data are sequentially merged to obtain the ciphertext data of the target file; Storing the ciphertext data of the target file; The method includes: determining a target application scenario corresponding to the target file; determining a preset sharding rule corresponding to the target application scenario from a plurality of preset sharding rules; dividing the plaintext data of the target file into a plurality of sharding data in sequence according to the preset sharding rule; and determining, in the process of dividing the sharding data, the associated parameters of the sharding data with respect to the target file according to the data features of the divided sharding data and the dividing order of the sharding data; The step of storing the ciphertext data of the target file includes: splitting the ciphertext data of the target file into multiple data components; storing the multiple data components in multiple data nodes in the HDFS distributed file system respectively, and recording storage information of the multiple data components; And, after storing the multiple data components in multiple data nodes in the HDFS distributed file system respectively, the method also includes: receiving a call request for the target file initiated by a terminal device; in response to the call request, sending the multiple data components of the ciphertext data of the target file and the splicing rules to the terminal device; wherein the terminal device is used to obtain the ciphertext data of the target file by locally splicing the multiple data components according to the splicing rules.

2. The method according to claim 1, characterized in that The data contained in the target file includes at least one of the following: transaction records of website users, account data of website users, and communication records of website users.

3. The method according to claim 2, characterized in that According to the preset fragmentation rules, the plaintext data of the target file is sequentially divided into multiple fragmentation data, including: According to the data characteristics of the target file, the plaintext data of the target file is divided into a plurality of fragmented data; wherein the data characteristics include at least one of the following: a source identifier of the data, a generation time of the data, and a type of the data.

4. The method according to claim 1, characterized in that: The preset encryption rules include encryption rules based on a homomorphic encryption algorithm.

5. The method according to claim 4, characterized in that Call multiple processing nodes in the HDFS distributed file system, use preset encryption rules, and encrypt the multiple shard data in parallel to obtain ciphertext data of the multiple shard data, including: Generate a public key and a private key corresponding to the target file according to a preset encryption rule; and distribute the public key and the plurality of shard data to a plurality of processing nodes in the HDFS distributed file system; The multiple processing nodes are called to encrypt the allocated shard data using the public key according to preset encryption rules to obtain ciphertext data of the multiple shard data.

6. The method according to claim 5, characterized in that Generate a public key and a private key corresponding to the target file according to a preset encryption rule, including: According to a preset encryption rule, randomly generate a first odd prime number, a second odd prime number, a first integer, and a second integer; generating an intermediate parameter according to the first odd prime number, the first integer and the second integer; The public key is determined according to the intermediate parameter and the product of the first odd prime number and the second odd prime number; and the first odd prime number is determined as a private key corresponding to the public key.

7. The method according to claim 1, characterized in that The call request also carries the identity of the target user corresponding to the terminal device.

8. The method according to claim 7, characterized in that The method further comprises: Extracting the identity of the target user from the call request; According to the identity identifier, verify the authority of the target user; When the target user authority verification is passed, the private key corresponding to the target file is sent to the terminal device; wherein the terminal device is used to decrypt the ciphertext data of the target file locally using the private key to obtain the plaintext data of the target file.

9. A file storage device, characterized in that: include: An acquisition module, used to acquire the plaintext data of a target file to be stored; wherein the target file is a data file with a large amount of data and which requires data privacy protection; A segmentation module, used to segment the plaintext data of the target file into a plurality of segment data in sequence according to a preset segmentation rule; A determination module, used to determine and set a corresponding association identifier for the shard data according to an association parameter of the shard data with respect to the target file; wherein the association parameter is used to characterize the association relationship between the shard data segmented based on a preset sharding rule and the target file; The calling module is used to call multiple processing nodes in the HDFS distributed file system, and encrypt the multiple shard data in parallel using preset encryption rules to obtain ciphertext data of the multiple shard data; A merging module, used to merge the ciphertext data of the plurality of fragmented data in order according to the association identifier to obtain the ciphertext data of the target file; A storage module, used for storing the ciphertext data of the target file; The segmentation module is specifically used to determine the target application scenario corresponding to the target file; determine a preset segmentation rule corresponding to the target application scenario from multiple preset segmentation rules; divide the plaintext data of the target file into multiple segmented data in sequence according to the preset segmentation rule; and in the process of segmenting the segmented data, determine the associated parameters of the segmented data with respect to the target file according to the data characteristics of the segmented segmented data and the segmentation order of the segmented data; The storage module is specifically used to split the ciphertext data of the target file into multiple data components; store the multiple data components in multiple data nodes in the HDFS distributed file system respectively, and record the storage information of the multiple data components; Furthermore, after storing the multiple data components in multiple data nodes in the HDFS distributed file system respectively, the apparatus is also used to: receive a call request for a target file initiated by a terminal device; in response to the call request, send multiple data components of the ciphertext data of the target file and a splicing rule to the terminal device; wherein the terminal device is used to obtain the ciphertext data of the target file by locally splicing the multiple data components according to the splicing rule.

10. A server, characterized in that: The method comprises a processor and a memory for storing processor-executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the instructions.

11. A computer storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed, the steps of the method according to any one of claims 1 to 8 are implemented.

Citation Information

Patent Citations

  • File fragmentation encryption storage method, acquisition method and system based on block chain

    CN111917720A