Data file encryption method and apparatus, computing device, and storage medium

Abstract

The application discloses a data file encryption method and device, a computing device and a storage medium. The method comprises the following steps: receiving a file uploading request of a data file, storing a plurality of data fragments of the data file in a plurality of data nodes in a distributed storage system respectively; obtaining load information of a data node in which any data fragment of the data file is stored; determining an object of performing encryption processing according to the load information of the data node in which the data fragment is stored; and if it is determined that the object of performing encryption processing is the data node in which the data fragment is stored, controlling the data node to perform encryption processing on the data fragment. Through the above method, only a part of computing resources needs to be dynamically reserved in the data node, and the data node can complete the encryption processing of the data fragment when the load of the data node is low, without loading the data fragment to an external computing node for encryption processing, so that network bandwidth resources can be saved, and the situation that the data node has too large throughput can be avoided.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a data file encryption method, apparatus, computing device, and storage medium. Background Technology

[0002] To ensure that files are error-free and tamper-free after being uploaded to a distributed storage system, they are typically encrypted using encryption algorithms. Currently, the industry commonly employs a storage-compute separation approach for data encryption. This means that storage nodes and compute nodes operate independently; storage nodes are dedicated to storing data, while compute nodes are dedicated to encryption calculations. Therefore, data is first retrieved from the storage system and then sent to the compute nodes through the storage system's gateway, where the compute nodes encrypt the retrieved data. The advantage of this approach is business decoupling. However, this approach also has several drawbacks, including lower encryption processing efficiency, higher network bandwidth consumption, and increased throughput on data nodes. Summary of the Invention

[0003] The purpose of this application is to provide a data file encryption method, apparatus, computing device, and storage medium to solve the problems of low encryption processing efficiency, high network bandwidth consumption, and increased throughput of data nodes caused by the above-mentioned storage-computation separation method.

[0004] According to one aspect of this application, a data file encryption method is provided, comprising:

[0005] Receive file upload requests for data files and store multiple data fragments of the data file into multiple data nodes in the distributed storage system.

[0006] Retrieve the load information of the data nodes stored in any data segment of the data file;

[0007] Based on the load information of the data nodes stored in the data shard, determine the object to be encrypted;

[0008] If it is determined that the object to be encrypted is the data node where the data shard is stored, control the data node to encrypt the data shard.

[0009] Optionally, after determining the object to be encrypted, the method further includes:

[0010] If the object to be encrypted is determined to be a compute node, the compute node is controlled to read the data fragment from the data node and load it locally on the compute node for encryption.

[0011] Optionally, before receiving a file upload request for a data file, the method further includes:

[0012] Receive multiple data fragments of a data file uploaded via a resume upload method;

[0013] Multiple data shards are generated by dividing the data file according to a first predetermined size.

[0014] Optionally, the load information includes: CPU load information and / or disk load information;

[0015] Based on the load information of the data nodes stored in the data shard, the objects to be encrypted further include:

[0016] If the CPU load information of the data node stored in the data shard is lower than the first preset value and / or the disk load information is lower than the second preset value, the object to be encrypted is determined to be the data node.

[0017] If the CPU load information of the data node in the data shard is higher than a first preset value and / or the disk load information is higher than a second preset value, the object to be encrypted is determined to be the computing node.

[0018] Optionally, controlling the data node to encrypt the data fragment further includes:

[0019] A first computing task is created and sent to the data node for the data node to run; the first computing task is used to encrypt the data shard.

[0020] Optionally, controlling the compute node to read the data shard from the data node and load it locally on the compute node for encryption further includes:

[0021] A second computing task is created and sent to the computing node for the computing node to run. The second computing task is used to read the data fragment from the data node, load it locally on the computing node, and encrypt the data fragment.

[0022] Optionally, encrypting the data fragment further includes:

[0023] Encryption is performed on the data of this data segment and the encryption result of the previous data segment to obtain the encryption result of this data segment;

[0024] If the data fragment is the first data fragment in the data file, the encryption result of the previous data fragment is the preset encryption result.

[0025] Optionally, if the data fragment is the first data fragment in the data file, encryption processing is performed based on the data of the data fragment and the encryption processing result of the previous data fragment. The encryption processing result of the data fragment further includes:

[0026] If the size of the data in the data segment is an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups; if the size of the data in the data segment is not an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups and tail data.

[0027] Encryption operations are performed based on multiple groups and the preset encryption processing results to obtain intermediate hash results.

[0028] Optionally, if the data fragment is any data fragment other than the first data fragment, the encryption process based on the data of the data fragment and the encryption result of the previous data fragment further includes:

[0029] If the encryption result of the previous data segment includes the intermediate hash result and the tail data, the tail data corresponding to the previous data segment is concatenated with the data of this data segment; the concatenated data and the intermediate hash result corresponding to the previous data segment are used for encryption to obtain the encryption result of this data segment.

[0030] If the encryption result of the previous data segment includes the intermediate hash result but does not include the tail data, then the encryption process is performed based on the data of this data segment and the intermediate hash result corresponding to the previous data segment to obtain the encryption result of this data segment.

[0031] Optionally, the method further includes:

[0032] Extract the intermediate hash result contained in the encryption processing result of the last data segment of the data file, and save the extracted intermediate hash result as the encryption processing result of the data file.

[0033] Optionally, after determining that the object to be encrypted is a computing node, the method further includes:

[0034] Based on the load information of each computing node in the computing node cluster, the target computing node is selected from the computing node cluster.

[0035] Controlling the compute node to read the data shard from the data node and load it locally on the compute node for encryption further includes:

[0036] The target compute node is controlled to read the data shard from the data node and load it locally on the compute node for encryption processing.

[0037] According to another aspect of this application, a data file encryption device is provided, comprising:

[0038] The acquisition module is suitable for receiving file upload requests for data files and storing multiple data fragments of the data file into multiple data nodes in the distributed storage system.

[0039] The decision module is adapted to obtain the load information of the data nodes stored in any data segment of the data file; and determine the object to be encrypted based on the load information of the data nodes stored in the data segment.

[0040] The processing module is adapted to control the data node to perform encryption processing on the data fragment if it is determined that the object to be encrypted is the data node where the data fragment is stored.

[0041] Optionally, the processing module is further adapted to: if it is determined that the object to be encrypted is a computing node, control the computing node to read the data fragment from the data node and load it locally on the computing node to encrypt the data fragment.

[0042] Optionally, the apparatus further includes: a file receiving module, adapted to receive multiple data fragments of a data file uploaded via a breakpoint resume method; wherein the multiple data fragments are generated by dividing the data file according to a first predetermined size.

[0043] Optionally, the load information includes: CPU load information and / or disk load information; the decision module is further adapted to:

[0044] If the CPU load information of the data node stored in the data shard is lower than the first preset value and / or the disk load information is lower than the second preset value, the object to be encrypted is determined to be the data node; if the CPU load information of the data node stored in the data shard is higher than the first preset value and / or the disk load information is higher than the second preset value, the object to be encrypted is determined to be the computing node.

[0045] Optionally, the processing module is further adapted to: create a first computing task and send the first computing task to the data node for the data node to run the first computing task; wherein the first computing task is used to encrypt the data fragment.

[0046] Optionally, the processing module is further adapted to: create a second computing task and send the second computing task to the computing node for the computing node to run the second computing task; wherein the second computing task is used to read the data fragment from the data node and load it locally on the computing node and encrypt the data fragment.

[0047] Optionally, the encryption process for the data fragment includes the following steps: performing encryption processing based on the data of the data fragment and the encryption processing result of the previous data fragment to obtain the encryption processing result of the data fragment; wherein, if the data fragment is the first data fragment in the data file, the encryption processing result of the previous data fragment is a preset encryption processing result.

[0048] Optionally, if the data fragment is the first data fragment in the data file, and if the size of the data fragment is an integer multiple of the second predetermined size, the data fragment is grouped according to the second predetermined size to obtain multiple groups; if the size of the data fragment is not an integer multiple of the second predetermined size, the data fragment is grouped according to the second predetermined size to obtain multiple groups and tail data; an encryption operation is performed based on the multiple groups and the preset encryption processing result to obtain a hash intermediate result.

[0049] Optionally, if the data shard is any data shard other than the first data shard, and the encryption result of the previous data shard includes a hash intermediate result and tail data, the tail data corresponding to the previous data shard is concatenated with the data of this data shard; the concatenated data and the hash intermediate result corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard; if the encryption result of the previous data shard includes a hash intermediate result but does not include tail data, the data of this data shard and the hash intermediate result corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard.

[0050] Optionally, the processing module is further adapted to: extract the intermediate hash result contained in the encryption processing result of the last data segment of the data file, and save the extracted intermediate hash result as the encryption processing result of the data file.

[0051] Optionally, the decision module is further adapted to: select a target computing node from the computing node cluster based on the load information of each computing node in the computing node cluster; the processing module is further adapted to: control the target computing node to read the data shard from the data node and load it locally on the computing node to encrypt the data shard.

[0052] According to another aspect of this application, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus;

[0053] The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-described data file encryption method.

[0054] According to another aspect of this application, a computer storage medium is provided, wherein the storage medium stores at least one executable instruction that causes a processor to perform operations corresponding to the data file encryption method described above.

[0055] According to the data file encryption method, apparatus, computing device, and storage medium of this application, in the scenario of distributed data file storage, only a portion of computing resources need to be dynamically reserved by the data nodes. When the data node load is low, the data nodes complete the encryption processing of data fragments without loading the data fragments to external computing nodes for encryption processing. This saves network bandwidth resources, improves the efficiency of data encryption processing, avoids excessive throughput of data nodes, improves the performance of data file encryption processing, and reduces the deployment of computing nodes. Furthermore, by creating an offline task for encryption computing, the encryption processing can be completed by controlling the data nodes or computing nodes to run computing tasks, thus improving the efficiency of data encryption. Furthermore, each data fragment of the data file is encrypted sequentially. The encryption processing result of the current data fragment is obtained by encrypting the data fragment based on the encryption processing result of the previous data fragment and the data of the current data fragment. The intermediate hash value in the encryption processing result of the last data fragment is extracted as the encryption result of the data file, which can obtain an accurate encryption result of the data file and solves the problem that the prior art cannot calculate the encryption result of data files in the distributed storage mode.

[0056] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0057] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0058] Figure 1A flowchart of the data file encryption method provided in an embodiment of this application is shown;

[0059] Figure 2 A flowchart of a data file encryption method provided in another embodiment of this application is shown;

[0060] Figure 3 The interactive flowchart of the data file encryption method according to an embodiment of this application is shown;

[0061] Figure 4 A schematic diagram of the structure of the data file encryption device provided in an embodiment of this application is shown;

[0062] Figure 5 A schematic diagram of the structure of a computing device provided in an embodiment of this application is shown. Detailed Implementation

[0063] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0064] First, the terms and concepts involved in one or more embodiments of this application will be explained.

[0065] Object storage systems combine the advantages of NAS and SAN, offering the high-speed direct access of SAN and the data sharing capabilities of NAS, providing a highly reliable, cross-platform, and secure data sharing storage architecture.

[0066] Data node: A node in a distributed storage system used to store data.

[0067] Computation node: A node that performs cryptographic computations.

[0068] Metadata cluster: A cluster of nodes used to store metadata. Metadata, also known as intermediary data or relay data, is data that describes data, mainly information that describes data attributes, and is used to support functions such as indicating storage location, historical data, resource lookup, and file records.

[0069] Resumable upload: When uploading a file to an object storage system, the file is split into multiple chunks of fixed size and uploaded concurrently. Each request uploads only a portion of the file. The upload process of different chunks is independent of each other, and the failure of some chunks will not affect other chunks that have already been uploaded.

[0070] MD5 Message-Digest Algorithm: MD5 is a widely used cryptographic hash function that produces a 128-bit (16-byte) hash value to ensure the integrity and consistency of transmitted information.

[0071] Distributed storage systems are systems that distribute data across multiple independent devices, offering advantages such as high reliability, high availability, high access efficiency, and easy scalability.

[0072] Data sharding: Data files are divided into shards according to a predetermined size.

[0073] Figure 1 A flowchart of a data file encryption method provided in an embodiment of this application is shown, such as... Figure 1 As shown, the method includes the following steps:

[0074] Step S110: Receive the file upload request for the data file and store the multiple data fragments of the data file into multiple data nodes in the distributed storage system.

[0075] The data file is divided into multiple data shards, which are then uploaded to the storage system. Specifically, Upload Part is used to upload all data shards. Once all data shards have been successfully uploaded, the CompleteMultipartUpload interface must be called to complete the shard upload of the entire file. When a file upload request is received, the multiple data shards are stored in multiple data nodes in the data node cluster of the distributed storage system.

[0076] Step S120: Obtain the load information of the data nodes stored in any data shard of the data file.

[0077] When encryption is required, for any data segment of the data file, the load of the data node where the data segment is stored is obtained, including CPU load and / or disk load, to assess whether the data node can still handle computing tasks.

[0078] Step S130: Determine the object to be encrypted based on the load information of the data nodes stored in the data shard.

[0079] Based on the load of the data node storing the data shard, assess whether the data node can still handle the computing task. If so, determine that the data node is the object to be encrypted.

[0080] Step S140: If it is determined that the object to be encrypted is a data node, control the data node to encrypt the data fragment.

[0081] If it is determined that the data node can still carry out computing tasks, then the data node is identified as the object to be encrypted. The data fragments are encrypted in the data node. Since the data fragments are stored in the data node, no network bandwidth is required, and the efficiency of encryption processing can be improved.

[0082] According to the data file encryption method of this embodiment, a file upload request for a data file is received, and multiple data fragments of the data file are stored in multiple data nodes in a distributed storage system. The load information of the data node where any data fragment of the data file is stored is obtained. Based on the load information of the data node where the data fragment is stored, the object to be encrypted is determined. If the object to be encrypted is determined to be the data node where the data fragment is stored, the data node is controlled to perform encryption processing on the data fragment. Through this method, in a distributed data file storage scenario, only a portion of computing resources needs to be dynamically reserved on the data node. When the load condition of the data node meets the requirements, the encryption processing of the data fragments is completed on the data node itself, without needing to load the data fragments to external computing nodes for encryption. This saves network resources, improves the efficiency of data encryption processing, avoids excessive throughput on data nodes, and enhances the performance of data file encryption processing.

[0083] Figure 2 A flowchart of a data file encryption method provided in another embodiment of this application is shown, such as... Figure 2 As shown, the method includes the following steps:

[0084] Step S210: Receive multiple data fragments of the data file uploaded via a breakpoint resume method; wherein the multiple data fragments are generated by dividing the data file according to a first predetermined size.

[0085] The first predetermined size can be a fixed setting or a custom setting by the business party. In the breakpoint resume mode, the data file is divided into multiple data chunks. For example, assuming the data file size is 35MB and the business party customizes the breakpoint resume chunk size to 10MB, the data file is divided into 4 data chunks during the upload process. The size of the first to third data chunks is 10MB each, and the size of the fourth data chunk is 5MB.

[0086] Step S220: Receive the file upload request for the data file and store the multiple data fragments of the data file into multiple data nodes in the distributed storage system.

[0087] After the various data fragments of the data file are uploaded to the distributed storage system, the business side calls CompleteMultipartUpload. Upon receiving this call request, the distributed storage system stores the multiple data fragments of the data file into multiple data nodes respectively. In this embodiment, the distributed storage system is built based on an object storage system.

[0088] Step S230: Obtain the load information of the data nodes stored in any data shard of the data file.

[0089] For any data segment of a data file, obtain the load information of the data node storing that data segment, specifically including CPU load information and / or disk load information.

[0090] Step S240: Determine the object to be encrypted based on the load information of the data nodes stored in the data shard.

[0091] The system assesses whether a data node can still handle computing tasks based on the load information of the data nodes stored in the data shard. Specifically, it compares the CPU load information of the data nodes stored in the data shard with a first preset value, and / or compares the disk load information of the data nodes stored in the data shard with a second preset value, and determines the object to be encrypted based on the comparison result.

[0092] If the CPU load information of the data node where the data shard is stored is lower than the first preset value or the disk load information is lower than the second preset value, the object to be encrypted is determined to be the data node. That is, when the load of the data node where the data shard is stored is low, the data node can still undertake the data encryption calculation task, and the object to be encrypted is determined to be the data node.

[0093] Conversely, if the CPU load information of the data node where the data shard is stored is higher than the first preset value or the disk load information is higher than the second preset value, that is, when the load of the data node where the data shard is stored is high, the data node can no longer undertake the additional computation work of data encryption. In this case, the object to be encrypted is determined to be the computing node. The computing node can be a node outside the distributed storage system used to perform computing tasks.

[0094] If the object to be encrypted is a data node, proceed to step S250; if the object to be encrypted is a computing node, proceed to step S260.

[0095] Step S250: Create a first computing task and send the first computing task to the data node so that the data node can run the first computing task.

[0096] If the object to be encrypted is determined to be a data node, an offline task for encryption computation, known as the first computation task, is created. This first computation task can be a script or code used to perform encryption computation. The first computation task is sent to the data node where the data shards are stored. The data node runs this first computation task to encrypt the data shards. Since the data shards are stored on the data node, the data node can directly read the data shards from locally stored files, thus eliminating the need for network bandwidth consumption and requiring only additional computing resources, thereby improving the speed of encryption computation. Furthermore, by creating an offline task for encryption computation and controlling the data node to run the computation task, the encryption process can be completed, improving the efficiency of data encryption.

[0097] Step S260: Control the compute node to read the data shard from the data node and load it locally on the compute node to encrypt the data shard.

[0098] If the object to be encrypted is determined to be a compute node, the compute node reads the data fragments from the data nodes and loads them locally, and then encrypts the data fragments loaded locally.

[0099] In one alternative approach, a second computation task is created and sent to a compute node for execution. This second computation task is also an offline task for encryption computation, and can be a script or code used to perform encryption processing. The compute node running the second computation task can perform the following operations: read data shards from the data node and load them locally, and then encrypt the loaded data shards. In other words, the encryption computation task is only sent to an external compute node when the data nodes are insufficient to handle it, thus reducing the deployment of compute nodes.

[0100] In one alternative approach, after encrypting the data shards, the encrypted information obtained from the encryption process is recorded in the metadata cluster of the distributed storage system, which facilitates the subsequent reading and management of the encrypted information of the data shards.

[0101] In one alternative approach, a compute node cluster is deployed. After determining that the object to be encrypted is a compute node, the target compute node is selected from the cluster based on the load information of each compute node. The target compute node then reads the data shard from the data node and loads it locally for encryption. This approach, by selecting a compute node from the cluster through load balancing to perform encryption, identifies the most efficient compute node for data encryption, thus improving the performance of the data encryption process.

[0102] It should be noted that the way compute nodes encrypt data shards is the same as the way data nodes encrypt data shards. The difference is that compute nodes need to load the data shards locally before they can perform encryption.

[0103] Specifically, encrypting the data fragment includes the following steps: Encrypting the data fragment based on its data and the encryption result of the preceding data fragment to obtain the encryption result of the data fragment; where, if the data fragment is the first data fragment in the data file, the encryption result of the preceding data fragment is a preset encryption result. That is, assuming the data file has a total of N data fragments, encrypting the (i-1)th data fragment and the data of the ith data fragment to obtain the encryption result corresponding to the ith data fragment, where 1 ≤ i ≤ N.

[0104] It should be noted that the encryption result of the data segment preceding the first data segment is a preset encryption result. This preset encryption result includes the intermediate hash result but does not include other tail data. Specifically, for the first data segment in the data file, if the size of the data segment is an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups; if the size of the data segment is an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups and tail data; encryption is performed based on the multiple groups and the preset encryption result to obtain the intermediate hash result. The encryption result of the first data segment has two cases: one is that it includes the intermediate hash result but does not include the tail data, and the other is that it includes both the intermediate hash result and the tail data.

[0105] Additionally, if the data shard is any data shard other than the first data shard, and the encryption result of the previous data shard includes both intermediate hash results and tail data, the tail data corresponding to the previous data shard is concatenated with the data of this data shard; the concatenated data and the intermediate hash results corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard; if the encryption result of the previous data shard includes intermediate hash results but does not include tail data, the data of this data shard and the intermediate hash results corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard.

[0106] The encryption process for the current data fragment differs depending on the encryption result of the previous data fragment. If the encryption result of the previous data fragment includes both intermediate hash results and tail data, then the tail data of the previous data fragment is concatenated with the data of the current data fragment, and encryption is performed based on the concatenated data and the intermediate hash results included in the encryption result of the previous data fragment. If the encryption result of the previous data fragment includes intermediate hash results but not tail data, then encryption is performed directly based on the intermediate hash results included in the encryption result of the previous data fragment and the data of the current data fragment. In other words, the data involved in the encryption process consists of two parts: the intermediate hash result of the previous data fragment and the data of the current data fragment, or the concatenation result of the current data fragment and the tail data of the previous data fragment, hereinafter referred to as the basic encryption data.

[0107] Specifically, the encrypted basic data is grouped according to a second predetermined size. If the encrypted basic data is an integer multiple of the second predetermined size, the encryption result includes the intermediate hash results obtained by encrypting each group. If the encrypted basic data is not an integer multiple of the second predetermined size, the encryption result includes the intermediate hash results obtained by encrypting multiple groups and tail data that is smaller than the second predetermined size.

[0108] Each data fragment is processed sequentially using the encryption method described above until the last data fragment is processed. In one optional approach, the intermediate hash result contained in the encryption processing result of the last data fragment of the data file is extracted, and this extracted intermediate hash result is saved as the encryption processing result of the data file. Furthermore, the encryption processing result of this data file is recorded in the metadata cluster of the distributed storage system.

[0109] In practical applications, the MD5 algorithm is frequently used in object storage systems. Object storage systems need to be able to calculate the MD5 hash of files and record it in the file's metadata. Currently, the industry-standard file verification method is to calculate the MD5 hash of each data shard, and then, during shard upload, perform an MD5 calculation on all shard MD5 digests to obtain a hash value used for verifying the entire file. This hash value is usually called an etag. When the application calls CompleteMultipartUpload, it can quickly return the etag, thus avoiding the problem of slow MD5 calculation caused by concurrent or out-of-order uploads of shards during interrupted uploads. However, in many application scenarios, the MD5 digest value is still needed to verify the correctness of the entire file. Therefore, for files with interrupted uploads, it is still necessary to retrieve the entire file and calculate its MD5 hash after the upload is complete.

[0110] The file encryption method in this application can be applied to calculate the MD5 hash value of a data file in a resume download mode. Figure 3 The diagram illustrates the interactive flowchart of a data file encryption method according to an embodiment of this application. The entire process includes the following steps:

[0111] First, the object storage system receives four shards of the data file: shard 1, shard 2, shard 3, and shard 4. The application calls `CompleteMultipartUpload`, and the backend, upon receiving this request, completes the file upload and returns a success message to the application. These four shards are stored on data nodes 1, 2, 3, and 4, respectively. The scheduling service is then notified to generate an offline MD5 calculation task. Next, the scheduling service is about to run the MD5 calculation task. First, it parses the location information of the shard storage nodes from the object storage system's metadata cluster.

[0112] The MD5 algorithm requires dividing the data to be encrypted into 512-bit segments. Each operation is performed by combining the 128-bit result from the previous round with the current 512-bit data. Therefore, if the size of the data to be encrypted is a multiple of 512 bits, the result after MD5 encryption will only contain a 128-bit hash intermediate result. If the size of the data to be encrypted is not a multiple of 512 bits, there will be a tail data of less than 512 bits after grouping the data into 512-bit segments. In this embodiment, this tail data is retained and not calculated, while the remaining data is MD5 encrypted to obtain a 128-bit hash intermediate result. That is, if the size of the data to be encrypted is not a multiple of 512 bits, the result after MD5 encryption will contain a tail data of less than 512 bits and a 128-bit hash intermediate result.

[0113] Specifically, the following steps are performed to calculate the MD5 hash value of the data file:

[0114] Step 1: The scheduling service determines that the load on data node 1 is low and sends an encryption command to data node 1;

[0115] Step 2: Data node 1 performs MD5 encryption on shard 1 to obtain a binary result {tail data, hash intermediate result};

[0116] Step 3: Data node 1 returns the encryption processing result to the scheduling service;

[0117] Step 4: The scheduling service determines that the load on data node 2 is low and sends an encryption command to data node 2;

[0118] Step 5: Data node 2 reads the tuple result of shard 1 {tail data, hash intermediate result} from the scheduling service;

[0119] Step 6: Data node 2 performs MD5 encryption calculation based on the tuple result of shard 1 and shard 2 to obtain the tuple result {tail data, hash intermediate result}.

[0120] Step 7: Data node 2 returns the encryption processing result to the scheduling service;

[0121] Step 8: The scheduling service determines that the load on data node 3 is low and sends an encryption command to data node 3;

[0122] Step 9: Data node 3 reads the tuple result of shard 2 {tail data, hash intermediate result} from the scheduling service;

[0123] Step 10: Data node 3 performs MD5 encryption calculation based on the tuple result of shard 2 and shard 3 to obtain the tuple result {tail data, hash intermediate result}.

[0124] Step 11: Data node 3 returns the encryption processing result to the scheduling service;

[0125] Step 12: The scheduling service determines that the load on data node 4 is low and sends an encryption command to data node 4;

[0126] Step 13: Data node 4 reads the tuple result of shard 3 from the scheduling service {tail data, hash intermediate result};

[0127] Step 14: Data node 4 performs MD5 encryption calculation based on the tuple result of shard 3 and shard 4 to obtain the tuple result {tail data, hash intermediate result}.

[0128] Step 15: Data node 4 returns the encryption processing result to the scheduling service.

[0129] Finally, the scheduling service sends a request to the metadata cluster of the object storage system to update the MD5 value of the data file. The MD5 value of the file will be set, and this MD5 value is the intermediate hash result contained in the encryption processing result returned by data node 4. Comparative verification shows that, compared to the existing method of calculating the MD5 value of distributed storage data files through an external computing cluster, the method described in this application reduces the time consumption for calculating the MD5 value of distributed storage data files by more than 50%.

[0130] According to the data file encryption method of this application embodiment, in the scenario of distributed data file storage, only a portion of computing resources need to be dynamically reserved by the data nodes. When the data node load is low, the data nodes complete the encryption processing of data fragments without loading the data fragments to external computing nodes for encryption processing. This saves network bandwidth resources, improves the efficiency of data encryption processing, avoids excessive throughput of data nodes, improves the performance of data file encryption processing, and reduces the deployment of computing nodes. Furthermore, by creating an offline task for encryption computing, the encryption processing can be completed by controlling the data nodes or computing nodes to run computing tasks, thus improving the efficiency of data encryption. Furthermore, each data fragment of the data file is encrypted sequentially. The encryption processing result of the current data fragment is obtained by encrypting the data fragment based on the encryption processing result of the previous data fragment and the data of the current data fragment. The intermediate hash value in the encryption processing result of the last data fragment is extracted as the encryption result of the data file, which can obtain an accurate encryption result of the data file and solves the problem that the prior art cannot calculate the encryption result of data files in the distributed storage mode.

[0131] Figure 4 A schematic diagram of the data file encryption device provided in this application embodiment is shown as follows: Figure 4 As shown, the device includes the following modules:

[0132] The acquisition module 41 is adapted to receive a file upload request for a data file and store multiple data fragments of the data file into multiple data nodes in a distributed storage system.

[0133] The decision module 42 is adapted to obtain the load information of the data nodes stored in any data segment of the data file; and determine the object to be encrypted based on the load information of the data nodes stored in the data segment.

[0134] Processing module 43 is adapted to control the data node to perform encryption processing on the data fragment if it is determined that the object to be encrypted is the data node where the data fragment is stored.

[0135] In an alternative embodiment, the processing module 43 is further adapted to: if it is determined that the object to be encrypted is a computing node, control the computing node to read the data fragment from the data node and load it locally on the computing node to encrypt the data fragment.

[0136] In one alternative embodiment, the apparatus further includes: a file receiving module adapted to receive multiple data fragments of a data file uploaded via a resume upload method; wherein the multiple data fragments are generated by dividing the data file according to a first predetermined size.

[0137] In one alternative approach, the load information includes: CPU load information and / or disk load information; the decision module 42 is further adapted to:

[0138] If the CPU load information of the data node stored in the data shard is lower than the first preset value and / or the disk load information is lower than the second preset value, the object to be encrypted is determined to be the data node; if the CPU load information of the data node stored in the data shard is higher than the first preset value and / or the disk load information is higher than the second preset value, the object to be encrypted is determined to be the computing node.

[0139] In an alternative approach, the processing module 43 is further adapted to: create a first computing task and send the first computing task to a data node for the data node to run the first computing task; wherein the first computing task is used to encrypt the data fragment.

[0140] In an alternative embodiment, the processing module 43 is further adapted to: create a second computing task and send the second computing task to a computing node for the computing node to run the second computing task; wherein the second computing task is used to read the data fragment from the data node and load it locally on the computing node and encrypt the data fragment.

[0141] In one optional approach, encrypting the data fragment specifically includes the following steps: encrypting the data of the data fragment and the encryption result of the previous data fragment to obtain the encryption result of the data fragment; wherein, if the data fragment is the first data fragment in the data file, the encryption result of the previous data fragment is a preset encryption result.

[0142] In one optional approach, if the data fragment is the first data fragment in the data file, and if the size of the data fragment is an integer multiple of a second predetermined size, the data fragment is grouped according to the second predetermined size to obtain multiple groups; if the size of the data fragment is not an integer multiple of the second predetermined size, the data fragment is grouped according to the second predetermined size to obtain multiple groups and tail data; an encryption operation is performed based on the multiple groups and the preset encryption processing result to obtain a hash intermediate result.

[0143] In one alternative approach, if the data shard is any data shard other than the first data shard, and the encryption result of the previous data shard includes a hash intermediate result and tail data, the tail data corresponding to the previous data shard is concatenated with the data of this data shard; the concatenated data and the hash intermediate result corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard; if the encryption result of the previous data shard includes a hash intermediate result but does not include tail data, the data of this data shard and the hash intermediate result corresponding to the previous data shard are then encrypted to obtain the encryption result of this data shard.

[0144] In an alternative embodiment, the processing module 43 is further adapted to: extract the intermediate hash result contained in the encryption processing result of the last data segment of the data file, and save the extracted intermediate hash result as the encryption processing result of the data file.

[0145] In an alternative approach, the decision module 42 is further adapted to: select a target computing node from the computing node cluster based on the load information of each computing node in the computing node cluster; the processing module 43 is further adapted to: control the target computing node to read the data shard from the data node and load it locally on the computing node to encrypt the data shard.

[0146] This application provides a non-volatile computer storage medium storing at least one executable instruction that can execute the data file encryption method in any of the above method embodiments.

[0147] Figure 5 The diagram shows a structural schematic of an embodiment of the computing device of this application. The specific embodiments of this application do not limit the specific implementation of the computing device.

[0148] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0149] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps in the above-described embodiment of the data file encryption method for computing devices.

[0150] Specifically, program 510 may include program code that includes computer operation instructions.

[0151] Processor 502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0152] Memory 506 is used to store program 510. Memory 506 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0153] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0154] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0155] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, features of the embodiments of this application are sometimes grouped together in a single embodiment, figure, or description thereof in the above description of exemplary embodiments of this application. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0156] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0157] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0158] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of this application. This application can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0159] It should be noted that the above embodiments are illustrative of this application and not restrictive, and those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A method for encrypting data files, comprising: Receive a file upload request for a data file, and store multiple data fragments of the data file into multiple data nodes in a distributed storage system; Obtain the load information of the data nodes stored in any data segment of the data file; Based on the load information of the data nodes stored in the data shard, determine the object to be encrypted; If the object to be encrypted is determined to be the data node where the data shard is stored, the data node is controlled to encrypt the data shard. If the data shard is any data shard other than the first data shard, and the encryption result of the preceding data shard includes a hash intermediate result and tail data, the tail data corresponding to the preceding data shard is concatenated with the data of this data shard. The encrypted base data, including the concatenated data and the hash intermediate result corresponding to the preceding data shard, is then encrypted to obtain the encryption result of the data shard. The encryption result of the data shard includes: a hash intermediate result obtained by encrypting multiple groups, or a hash intermediate result obtained by encrypting multiple groups and tail data, wherein the multiple groups and tail data are obtained by grouping the encrypted base data.

2. The method according to claim 1, wherein, After determining the object to be encrypted, the method further includes: If the object to be encrypted is determined to be a computing node, the computing node is controlled to read the data fragment from the data node and load it locally to encrypt the data fragment.

3. The method according to claim 1, wherein, Before receiving the file upload request for the data file, the method further includes: Receive multiple data fragments of a data file uploaded via a resume upload method; The multiple data shards are generated by dividing the data file according to a first predetermined size.

4. The method according to claim 1, wherein, The load information includes: CPU load information and / or disk load information; The step of determining the object to be encrypted based on the load information of the data nodes stored in the data shard further includes: If the CPU load information of the data node stored in the data shard is lower than a first preset value and / or the disk load information is lower than a second preset value, the object to be encrypted is determined to be the data node. If the CPU load information of the data node in the data shard is higher than a first preset value and / or the disk load information is higher than a second preset value, the object to be encrypted is determined to be the computing node.

5. The method according to any one of claims 1-4, wherein, The step of controlling the data node to encrypt the data fragment further includes: A first computing task is created and sent to the data node for the data node to run; wherein the first computing task is used to encrypt the data shard.

6. The method according to claim 2, wherein, The step of controlling the computing node to read the data fragment from the data node and load it locally on the computing node for encryption processing of the data fragment further includes: A second computing task is created and sent to a computing node for the computing node to run the second computing task; wherein, the second computing task is used to read the data fragment from the data node, load it into the local computing node, and encrypt the data fragment.

7. The method according to any one of claims 1-4, wherein, The encryption process for the data fragments further includes: Encryption is performed on the data of this data segment and the encryption result of the previous data segment to obtain the encryption result of this data segment; If the data fragment is the first data fragment in the data file, the encryption result of the previous data fragment is the preset encryption result.

8. The method according to claim 7, wherein, If the data fragment is the first data fragment in the data file, encryption processing is performed based on the data of the data fragment and the encryption processing result of the previous data fragment. The encryption processing result of the data fragment further includes: If the size of the data in the data segment is an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups; if the size of the data in the data segment is not an integer multiple of the second predetermined size, the data segment is grouped according to the second predetermined size to obtain multiple groups and tail data. Encryption operations are performed based on the multiple groups and the preset encryption processing results to obtain intermediate hash results.

9. The method according to claim 7, wherein, If the data fragment is any data fragment other than the first data fragment, the encryption process based on the data of the data fragment and the encryption result of the previous data fragment further includes: If the encryption result of the previous data segment includes the intermediate hash result but does not include the tail data, the encryption process is performed based on the data of the data segment and the intermediate hash result corresponding to the previous data segment to obtain the encryption result of the data segment.

10. The method according to claim 9, wherein, The method further includes: Extract the intermediate hash result contained in the encryption processing result of the last data segment of the data file, and save the extracted intermediate hash result as the encryption processing result of the data file.

11. The method according to claim 2, wherein, After determining that the object to be encrypted is a computing node, the method further includes: Based on the load information of each computing node in the computing node cluster, the target computing node is selected from the computing node cluster. The step of controlling the computing node to read the data fragment from the data node and load it locally on the computing node for encryption processing of the data fragment further includes: The target computing node is controlled to read the data fragment from the data node and load it locally on the target computing node for encryption processing.

12. A data file encryption device, comprising: The acquisition module is adapted to receive file upload requests for data files and store multiple data fragments of the data file into multiple data nodes in a distributed storage system. The decision module is adapted to obtain the load information of the data nodes stored in any data segment of the data file; and determine the object to be encrypted based on the load information of the data nodes stored in the data segment. The processing module is adapted to control the data node to perform encryption processing on the data segment if it is determined that the object to be encrypted is the data node where the data segment is stored; wherein, if the data segment is any data segment other than the first data segment, and the encryption processing result of the previous data segment includes a hash intermediate result and tail data, the tail data corresponding to the previous data segment is concatenated with the data of the data segment; the encryption base data containing the concatenated data and the hash intermediate result corresponding to the previous data segment is encrypted to obtain the encryption processing result of the data segment; the encryption processing result of the data segment includes: a hash intermediate result obtained by encrypting multiple groups, or a hash intermediate result obtained by encrypting multiple groups and tail data, wherein the multiple groups and tail data are obtained by grouping the encryption base data.

13. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the data file encryption method as described in any one of claims 1-11.

14. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the data file encryption method as described in any one of claims 1-11.

15. A computer program product comprising at least one executable instruction that causes a processor to perform an operation corresponding to the data file encryption method as described in any one of claims 1-11.

Images