Fragmentation storage method, electronic equipment and computer readable medium

Abstract

The invention discloses a fragment storage method, electronic equipment and a computer readable medium. The method comprises the following steps: generating a global unique identifier for a to-be-stored file; the to-be-stored file comprises fragmented files; for the current to-be-stored fragmented file, generating a fragmented file name and a target storage directory for the current to-be-stored fragmented file according to the global unique identifier, the storage path of each current storage device, a preset hash algorithm and the fragmentation sequence identifier of the current to-be-stored fragmented file; obtaining encryption demand information of the current to-be-stored fragmented file; and according to the fragment file name and the encryption demand information, storing the current to-be-stored fragment file in the target storage directory. According to the technical scheme, the attributes of the fragmented files and the incidence relation between the fragmented files are hidden, the safety of data can be effectively improved, and different encryption requirements are applied to different fragmented files, so that the storage cost can be reduced, the storage efficiency can be improved, and the storage experience of a user can be improved.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular, to a shard storage method, an electronic device, and a computer-readable medium. Background Art

[0002] Currently, storage systems generally use sharded storage to replace traditional overall storage when storing large files. This is because sharded storage can not only optimize resource utilization and improve storage system performance, but also support dynamic expansion and flexible management of storage systems.

[0003] The shard storage method usually uses the naming rule of "unified prefix + shard ID" to name the shard files, and encrypts the shard files using hardware encryption or software encryption, and stores all shard files in the same directory.

[0004] However, although the naming rule of "unified prefix + shard ID" is adopted and all shard files are stored in the same directory, although it is possible to more intuitively identify which shard files belong to the same large file, the relationship between the shard files is obvious and the file attributes are also exposed, which is not conducive to data security protection; at the same time, the use of software encryption often requires a large amount of CPU resources, which will lead to a decrease in upload and download rates in concurrent situations, seriously affecting the user experience; and the use of hardware encryption, although it can improve the security of user data and ensure user experience, the storage system using hardware encryption is relatively expensive.

[0005] Therefore, a new shard storage method is urgently needed. Summary of the invention

[0006] The present application aims to solve one of the technical problems in the related art to a certain extent. To this end, the present application provides a sharding storage method, an electronic device and a computer-readable medium.

[0007] As a first aspect of the present application, a shard storage method is provided, wherein the method comprises:

[0008] Generate a globally unique identifier for a file to be stored; wherein the file to be stored includes a fragment file;

[0009] For the shard file to be stored currently, generate a shard file name and a target storage directory for the shard file to be stored currently according to the globally unique identifier, the storage paths of the current storage devices, the preset hash algorithm, and the shard sequence identifier of the shard file to be stored currently;

[0010] Obtain encryption requirement information of the slice file to be stored currently;

[0011] According to the segment file name and the encryption requirement information, the segment file currently to be stored is stored in the target storage directory.

[0012] Optionally, generating a shard file name and a target storage directory for the shard file to be stored according to the globally unique identifier, the current storage paths of each storage device, a preset hash algorithm, and a shard sequence identifier of the shard file to be stored, includes:

[0013] Generate a fragment file name for the fragment file to be stored currently according to the globally unique identifier, the preset hash algorithm, and the fragment sequence identifier of the fragment file to be stored currently;

[0014] Determine a target storage device from among the current storage devices according to the fragment sequence identifier of the fragment file to be stored currently;

[0015] According to the storage path of the target storage device and the slice file name of the slice file to be stored currently, a target storage directory is generated for the slice file to be stored currently.

[0016] Optionally, generating a target storage directory for the slice file to be stored currently according to the storage path of the target storage device and the slice file name of the slice file to be stored currently includes:

[0017] Generate a subdirectory according to the data located at a preset position in the name of the segment file of the segment file to be stored currently; wherein the subdirectory is a three-level directory;

[0018] According to the storage path of the target storage device and the subdirectory, a target storage directory is generated for the slice file currently to be stored.

[0019] Optionally, the obtaining encryption requirement information of the current shard file to be stored includes:

[0020] Determining a storage encryption strategy for the file to be stored from a plurality of preset storage encryption strategies according to the attribute information of the file to be stored;

[0021] The encryption requirement information of the current segmented file to be stored is determined at least according to the storage encryption policy of the file to be stored.

[0022] Optionally, the multiple preset storage encryption strategies include a small file full encryption strategy, a text file full encryption strategy, a large file first fragment encryption strategy, and a large file percentage encryption strategy; the step of determining the encryption storage strategy of the file to be stored from the multiple preset encryption storage strategies according to the attribute information of the file to be stored includes:

[0023] When it is determined according to the attribute information of the file to be stored that the size of the file to be stored does not exceed a preset threshold, the small file full encryption policy is determined as the encryption storage policy for the file to be stored;

[0024] In the case where it is determined according to the attribute information of the file to be stored that the type of the file to be stored is a text file, the full encryption policy of the text file is determined as the encryption storage policy of the file to be stored;

[0025] When it is determined based on the attribute information of the file to be stored that the size of the file to be stored exceeds the preset threshold and the type is not a text file, the large file first fragment encryption strategy and the large file percentage encryption strategy are displayed, an externally input policy indication message is received, and the encryption storage policy of the file to be stored indicated in the policy indication message is obtained.

[0026] Optionally, the step of determining encryption requirement information of the current shard file to be stored at least according to a storage encryption policy of the file to be stored includes:

[0027] When the storage encryption policy of the file to be stored is the small file full encryption policy or the text file full encryption policy, determining that the encryption requirement information of the current shard file to be stored is encryption required;

[0028] In a case where the storage encryption policy of the file to be stored is the first shard encryption policy of the large file, determining encryption requirement information of the shard file to be stored according to the shard sequence identifier of the shard file to be stored currently;

[0029] When the storage encryption policy of the file to be stored is the large file percentage encryption policy, the encryption requirement information of the slice file to be stored is determined according to the slice sequence identifier of the slice file to be stored, the total number of slice files in the file to be stored, and the percentage information indicated in the large file percentage encryption policy.

[0030] Optionally, the encryption requirement information of the shard file to be stored currently is whether encryption is required or not, and storing the shard file to be stored currently in the target storage directory according to the shard file name and the encryption requirement information includes:

[0031] In the case where the encryption requirement information of the currently to-be-stored segment file is that encryption is required, after encrypting the currently to-be-stored segment file according to a preset encryption algorithm, the encrypted segment file is stored in the target storage directory according to the segment file name;

[0032] When the encryption requirement information of the segment file to be stored currently indicates that encryption is not required, the segment file to be stored currently is stored in the target storage directory according to the segment file name.

[0033] Optionally, the preset encryption algorithm is any one of the following:

[0034] XOR encryption algorithm, Data Encryption Standard DES algorithm, Triple Data Encryption Standard 3DES algorithm, Advanced Encryption Standard AES algorithm.

[0035] As a second aspect of the present application, an electronic device is provided, wherein the electronic device includes:

[0036] one or more processors;

[0037] A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the sharding storage method according to the first aspect of the present application.

[0038] As a third aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the shard storage method according to the first aspect of the present application is implemented.

[0039] In the shard storage method provided in the embodiment of the present application, a globally unique identifier is generated for the file to be stored, and the file to be stored includes a shard file. For the shard file currently to be stored, a shard file name and a target storage directory are generated for the shard file currently to be stored according to the globally unique identifier, the storage path of each current storage device, a preset hash algorithm, and the shard sequence identifier of the shard file currently to be stored. This is equivalent to hiding the attributes of the shard file and the association relationship between the shard files by blurring the name of the shard file and the storage directory, which can effectively improve the security of the data. Furthermore, the encryption requirement information of the shard file currently to be stored is obtained, and according to the shard file name and the encryption requirement information, the shard file currently to be stored is stored in the target storage directory. By applying different encryption requirements to different shard files, it is also possible to reduce storage costs, improve storage efficiency, and improve user storage experience while effectively improving data security.

[0040] For storage systems of personal users, in order to balance cost saving and practicality, high-performance storage devices are usually not used. However, even if low-performance storage devices are used, better data security protection can be obtained by using the shard storage method provided by the embodiment of the present application, thereby reducing the cost of storage devices. In addition, the storage system of personal users mainly stores large files such as audio, video, and images of the user's family life. By using the shard storage method provided by the embodiment of the present application, different encryption requirements are imposed on different shard files, which effectively improves the security of the data without reducing the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present application is further described below in conjunction with the accompanying drawings:

[0042] Figure 1 It is a flowchart of an implementation method of the shard storage method provided in an embodiment of the present application;

[0043] Figure 2 This is a flowchart of an implementation method of generating a shard file name and a target storage directory provided in an embodiment of the present application;

[0044] Figure 3 This is a flowchart of an implementation method of generating a target storage directory provided in an embodiment of the present application;

[0045] Figure 4 It is a flowchart of an implementation method of obtaining encryption requirement information of a shard file to be stored provided in an embodiment of the present application;

[0046] Figure 5 It is a flowchart of another implementation of the shard storage method provided in an embodiment of the present application;

[0047] Figure 6 It is a flow chart of an implementation of an electronic device provided in an embodiment of the present application;

[0048] Figure 7 It is a schematic diagram of a computer-readable medium provided in an embodiment of the present application.

[0049] Description of Reference Numerals

[0050] 101: Processor 102: Memory

[0051] 103: I / O interface 104: bus DETAILED DESCRIPTION

[0052] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments based on the embodiments are intended to be used to explain the present application and cannot be understood as limiting the present application.

[0053] References to "one embodiment" or "an example" or "an example" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment itself may be included in at least one embodiment disclosed in the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.

[0054] Currently, storage systems generally use sharded storage to replace traditional overall storage when storing large files. This is because sharded storage can not only optimize resource utilization and improve storage system performance, but also support dynamic expansion and flexible management of storage systems.

[0055] The shard storage method usually uses the naming rule of "unified prefix + shard ID" to name the shard files, and uses hardware encryption or software encryption to encrypt the shard files, and stores all shard files in the same directory. Software encryption is widely used in the field of storage encryption. Common software encryption algorithms include Data Encryption Standard (DES) algorithm, Triple Data Encryption Standard (3DES) algorithm, Advanced Encryption Standard (AES) algorithm, etc. This encryption method can improve data security to a certain extent and protect users' sensitive information. The hardware encryption method has higher performance and security. Common hardware encryption devices include encrypted self-encrypting drives (SED), hardware security modules (HSM), full disk encryption (FDE), etc.

[0056] However, although the naming rule of "unified prefix + shard ID" is adopted and all shard files are stored in the same directory, although it is possible to more intuitively identify which shard files belong to the same large file, the relationship between the shard files is obvious and the file attributes are also exposed, which is not conducive to data security protection; at the same time, the use of software encryption often requires a lot of CPU resources, and the upload and download rates will drop in concurrent situations, seriously affecting the user experience; and the use of hardware encryption can improve the security of user data and ensure user experience, but the storage system using hardware encryption is relatively expensive.

[0057] In view of this, the applicant of this application proposes that hiding the association between shard files and the attributes of shard files can effectively improve data security. By imposing different encryption requirements on different shard files, the performance of the storage system can be improved while improving data security, thereby improving user experience.

[0058] As a first aspect of an embodiment of the present application, a shard storage method is provided, wherein: Figure 1 As shown, the method may include:

[0059] In step S110, a globally unique identifier is generated for the file to be stored; wherein the file to be stored includes a fragment file;

[0060] In step S120, for the shard file to be stored currently, a shard file name and a target storage directory are generated for the shard file to be stored currently according to the globally unique identifier, the storage paths of the current storage devices, the preset hash algorithm, and the shard sequence identifier of the shard file to be stored currently;

[0061] In step S130, the encryption requirement information of the current segment file to be stored is obtained;

[0062] In step S140, the segmented file to be stored is stored in the target storage directory according to the segmented file name and the encryption requirement information.

[0063] The files to be stored have a globally unique identifier, that is, the globally unique identifiers of any two files to be stored are different, so that different files to be stored can be distinguished.

[0064] Among them, in the embodiment of the present application, there is no special limitation on the number of segment files included in the file to be stored, that is, the file to be stored can be stored as a whole as a segment file, or can be stored separately as multiple segment files.

[0065] Among them, the preset hash algorithm in the embodiment of the present application is not particularly limited. For example, the preset hash algorithm can be the Message Digest Algorithm Version 5 (Message Digest Algorithm, MD5), the Secure Hash Algorithm 256 (Secure Hash Algorithm-256, SHA-256), the Cyclic Redundancy Check algorithm (Cyclic Redundancy Check, CRC32), etc.

[0066] Among them, in the embodiment of the present application, there is no special limitation on how to set the fragment sequence identifier of the fragment file, as long as the fragment sequence identifier can be used to distinguish and locate all fragment files of the stored file. For example, if PartId represents the fragment sequence identifier, PartId can take the values ​​0, 1, 2,...n in sequence.

[0067] As described above, the embodiment of the present application does not specifically limit the execution order between the above steps S110-S120 and the above step S130, and they can be executed in parallel or one after another.

[0068] In the shard storage method provided in the embodiment of the present application, a globally unique identifier is generated for the file to be stored, and the file to be stored includes a shard file. For the shard file currently to be stored, a shard file name and a target storage directory are generated for the shard file currently to be stored according to the globally unique identifier, the storage path of each current storage device, a preset hash algorithm, and the shard sequence identifier of the shard file currently to be stored. This is equivalent to hiding the attributes of the shard file and the association relationship between the shard files by blurring the name of the shard file and the storage directory, which can effectively improve the security of the data. Furthermore, the encryption requirement information of the shard file currently to be stored is obtained, and according to the shard file name and the encryption requirement information, the shard file currently to be stored is stored in the target storage directory. By applying different encryption requirements to different shard files, it is also possible to reduce storage costs, improve storage efficiency, and improve user storage experience while effectively improving data security.

[0069] For storage systems of personal users, in order to balance cost saving and practicality, high-performance storage devices are usually not used. However, even if low-performance storage devices are used, better data security protection can be obtained by using the shard storage method provided by the embodiment of the present application, thereby reducing the cost of storage devices. In addition, the storage system of personal users mainly stores large files such as audio, video, and images of the user's family life. By using the shard storage method provided by the embodiment of the present application, different encryption requirements are imposed on different shard files, which effectively improves the security of the data without reducing the user's experience.

[0070] In some embodiments, the shard file name and the target storage directory are generated for the shard file to be stored according to the globally unique identifier, the current storage path of each storage device, the preset hash algorithm, and the shard sequence identifier of the shard file to be stored (i.e., step S120), such as Figure 2 As shown, it may include:

[0071] In step S121, a fragment file name is generated for the fragment file to be stored currently according to the globally unique identifier, the preset hash algorithm and the fragment sequence identifier of the fragment file to be stored currently;

[0072] In step S122, a target storage device is determined from among the current storage devices according to the fragment sequence identifier of the fragment file to be stored currently;

[0073] In step S123, a target storage directory is generated for the slice file to be stored currently according to the storage path of the target storage device and the slice file name of the slice file to be stored currently.

[0074] Among them, FileId, PartId, and PartFileName respectively represent the globally unique identifier of the file to be stored, the fragment sequence identifier of the fragment file currently to be stored, and the fragment file name of the fragment file currently to be stored. The PartFileName can be obtained by calculating FileId+PartId using the hash algorithm, which can be abstractly expressed as: PartFileName=Hash(FileId+PartId).

[0075] Among them, the embodiment of the present application does not make any special limitation on how to execute step S122, as long as it can ensure that each shard file is stored in each storage device in a dispersed manner. For example, when there are currently 2 storage devices, a certain file to be stored has two shard files, and the shard sequence identifier PartId is 1 and 2 respectively, for the shard file with PartId=1, the first storage device is selected as its target storage device, and for the shard file with PartId=2, the second storage device is selected as its target storage device.

[0076] The embodiment of the present application does not specifically limit how to execute step S123. For example, a subdirectory under the storage path of the target storage device may be generated according to the slice file name of the slice file to be stored. Accordingly, in some embodiments, the target storage directory (i.e., step S123) is generated for the slice file to be stored according to the storage path of the target storage device and the slice file name of the slice file to be stored. Figure 3 As shown, it may include:

[0077] In step S1231, a subdirectory is generated according to the data located at a preset position in the name of the segment file of the segment file to be stored currently; wherein the subdirectory is a three-level directory;

[0078] In step S1232, a target storage directory is generated for the slice file to be stored currently according to the storage path of the target storage device and the subdirectory.

[0079] The embodiment of the present application does not specifically limit the preset position, and the first N digits, the last N digits, or the Mth to Nth digits in the name of the segment file can be used as the preset position (M and N are both positive integers). For example, assuming that the name of the segment file is "f38ea124661da6497abac26b487b66e3" and the preset position is the first 6 digits of the name of the segment file, then the data at the preset position is "f38ea1", and the generated subdirectory is " / f3 / 8e / a1".

[0080] Among them, the target storage directory can be obtained by placing the generated subdirectory under the storage path of the target storage device. For example, assuming that the storage path of the target storage device is " / mnt / sda3 / data" and the generated subdirectory is " / f3 / 8e / a1", the generated target storage directory is " / mnt / sda3 / data / f3 / 8e / a1 / ".

[0081] The following is a detailed description of the process of generating a shard file name and a target storage directory (i.e., step S120) for the shard file to be stored according to the globally unique identifier, the storage paths of the current storage devices, the preset hash algorithm, and the shard sequence identifier of the shard file to be stored in the embodiment of the present application in combination with a most specific embodiment:

[0082] Assume that there are currently two storage devices, the storage paths are " / mnt / sda3 / data" and " / mnt / sdb3 / data", the preset hash algorithm is the MD5 algorithm, the preset position is the first 6 digits of the fragment file name, and there are a total of two fragment files to be stored.

[0083] First, a globally unique identifier FileId=1721360483190SY006 is generated for the file to be stored.

[0084] Furthermore, for the first fragment file, its PartId=1. Calculate the fragment file name PartFileName=Md5(1721360483190SY006+1)=f38ea124661da6497abac26b487b66e3. Select the first storage device as its target storage device. Take the first 6 characters of its fragment file name "f38ea1" to generate the subdirectory " / f3 / 8e / a1". Generate the target storage directory " / mnt / sda3 / data / f3 / 8e / a1 / ".

[0085] Furthermore, for the second fragment file, its PartId=2. Calculate the fragment file name PartFileName=Md5(1721360483190SY006+2)=60e2a5ef6f730938e9c87602c5c6aabf. Select the second storage device as its target storage device. Take the first 6 characters of its fragment file name "60e2a5" to generate the subdirectory " / 60 / e2 / a5". Generate the target storage directory " / mnt / sdb3 / data / 60 / e2 / a5 / ".

[0086] It can be seen that, using the shard storage method provided in the embodiment of the present application, even for the same file to be stored, the shard file names and the storage directories in which the different shard files are located are very different, so that the attributes of the shard files and the associations between the shard files are hidden, which can improve data security.

[0087] In the embodiment of the present application, the encryption requirement information of the shard file can be determined according to the storage encryption strategy used by the file to be stored. Accordingly, in some embodiments, the encryption requirement information of the shard file to be stored (i.e., step S130) is obtained, such as Figure 4 As shown, it may include:

[0088] In step S131, according to the attribute information of the file to be stored, a storage encryption strategy of the file to be stored is determined from a plurality of preset storage encryption strategies;

[0089] In step S132, encryption requirement information of the current segmented file to be stored is determined at least according to the storage encryption policy of the file to be stored.

[0090] It can be understood that the attribute information of a file may generally include the size of the file, the type of the file, and the like.

[0091] As described above, the embodiment of the present application does not specifically limit the execution order between the above steps S110-S120 and the above step S130. For example, for step S131, it can be executed before steps S110-S120, that is, first determine the encryption storage strategy of the file to be stored, then slice it according to the encryption storage strategy of the file to be stored to obtain one or more slice files, then generate a global unique identifier for the file to be stored, and then generate a slice file name and a target storage directory for each slice file; it can also be executed after step S110 and before step S120, that is, first generate a global unique identifier for the file to be stored, then determine the encryption storage strategy of the file to be stored, then slice it according to the encryption storage strategy of the file to be stored to obtain one or more slice files, and then generate a slice file name and a target storage directory for each slice file. For step S132, it can be executed synchronously with step S120, or it can be executed before step S120, or it can be executed after step S120.

[0092] Accordingly, in some embodiments, the plurality of preset storage encryption strategies include a small file full encryption strategy, a text file full encryption strategy, a large file first fragment encryption strategy, and a large file percentage encryption strategy. The step of determining the encryption storage strategy of the file to be stored from the plurality of preset encryption storage strategies according to the attribute information of the file to be stored (i.e., step S132) may include:

[0093] When it is determined according to the attribute information of the file to be stored that the size of the file to be stored does not exceed a preset threshold, the small file full encryption policy is determined as the encryption storage policy for the file to be stored;

[0094] In the case where it is determined according to the attribute information of the file to be stored that the type of the file to be stored is a text file, the full encryption policy of the text file is determined as the encryption storage policy of the file to be stored;

[0095] When it is determined based on the attribute information of the file to be stored that the size of the file to be stored exceeds the preset threshold and the type is not a text file, the large file first fragment encryption strategy and the large file percentage encryption strategy are displayed, an externally input policy indication message is received, and the encryption storage policy of the file to be stored indicated in the policy indication message is obtained.

[0096] Among them, the small file full encryption strategy means that for files to be stored whose size does not exceed a preset threshold, they can be stored as a fragment file as a whole, and the fragment file needs to be encrypted to improve data security.

[0097] Among them, the full encryption strategy for text files means that for files to be stored that are text files, since the text content is visible when any segment file is opened, all segment files need to be encrypted to improve data security.

[0098] Among them, the first-shard encryption strategy for large files means that for files to be stored whose size exceeds a preset threshold and whose type is not a text file, only the first-shard file can be encrypted, thereby improving data security and storage efficiency.

[0099] Among them, the large file percentage encryption strategy means that for files to be stored whose size exceeds the preset threshold and whose type is not a text file, only part of the fragmented files can be encrypted according to the percentage information indicated in the large file percentage encryption strategy, thereby improving data security and storage efficiency.

[0100] It can be seen that for files to be stored whose sizes exceed the preset threshold and whose types are not text files, they are regarded as large files, and either the large file first slice encryption strategy or the large file percentage encryption strategy can be adopted. In the embodiment of the present application, for such large files, the large file first slice encryption strategy and the large file percentage encryption strategy are displayed, and the user inputs a policy indication message from the outside to indicate which of the first slice encryption strategy and the large file percentage encryption strategy to use for the stored file. Since the data storage encryption strategy is selected by the user independently, it can well balance data protection and storage efficiency.

[0101] Among them, the embodiments of the present application do not make specific limitations on the preset threshold value of the file size, the percentage information of the shard file, etc. involved in the preset storage encryption strategy, and they can all be set according to actual conditions.

[0102] Accordingly, in some embodiments, determining the encryption requirement information of the current shard file to be stored (i.e., step S132) at least according to the storage encryption policy of the file to be stored may include:

[0103] When the storage encryption policy of the file to be stored is the small file full encryption policy or the text file full encryption policy, determining that the encryption requirement information of the current shard file to be stored is encryption required;

[0104] In a case where the storage encryption policy of the file to be stored is the first shard encryption policy of the large file, determining encryption requirement information of the shard file to be stored according to the shard sequence identifier of the shard file to be stored currently;

[0105] When the storage encryption policy of the file to be stored is the large file percentage encryption policy, the encryption requirement information of the slice file to be stored is determined according to the slice sequence identifier of the slice file to be stored, the total number of slice files in the file to be stored, and the percentage information indicated in the large file percentage encryption policy.

[0106] As described above, for files to be stored using the small file full encryption strategy and the text file full encryption strategy, each shard file needs to be encrypted, so the shard file currently to be stored must be encrypted. For files to be stored using the large file first shard encryption strategy, it is necessary to determine whether the shard file currently to be stored is the first shard of the file to be stored, so whether the shard file needs to be encrypted is determined based on the shard sequence identifier. For files to be stored using the large file percentage encryption strategy, it is necessary to determine whether the shard file currently to be stored falls within the percentage indicated by the large file percentage encryption strategy, so whether the shard file needs to be encrypted is determined based on the shard sequence identifier, the total number of shard files, and the percentage information indicated in the large file percentage encryption strategy.

[0107] In some embodiments, the encryption requirement information of the currently to-be-stored segmented file is whether encryption is required or not, and according to the segmented file name and the encryption requirement information, the currently to-be-stored segmented file is stored in the target storage directory (i.e., step S140), such as Figure 5 As shown, it may include:

[0108] In step S141, when the encryption requirement information of the segment file to be stored currently indicates that encryption is required, the segment file to be stored currently is encrypted according to a preset encryption algorithm, and the encrypted segment file is stored in the target storage directory according to the segment file name;

[0109] In step S142, when the encryption requirement information of the segment file to be stored currently indicates that encryption is not required, the segment file to be stored currently is stored in the target storage directory according to the segment file name.

[0110] For example, after storage is completed, the fragment file exists as a file named "60e2a5ef6f730938e9c87602c5c6aabf" under the storage directory " / mnt / sdb3 / data / 60 / e2 / a5 / ".

[0111] In some embodiments, the preset encryption algorithm is any one of the following: XOR encryption algorithm, Data Encryption Standard DES algorithm, Triple Data Encryption Standard 3DES algorithm, Advanced Encryption Standard AES algorithm.

[0112] It is understandable that, in addition to the above encryption algorithms, any other algorithm that can encrypt data can also be used.

[0113] In some embodiments, the preset hash algorithm may be pre-set by a user, and the method may further include: receiving an externally input hash algorithm indication message, and acquiring the preset hash algorithm indicated in the hash algorithm indication algorithm.

[0114] Among them, the user inputs a hash algorithm indication message from the outside to set the hash algorithm used according to the importance of the file. Since the encryption algorithm of the data can be selected by the user independently, it can well balance data protection and storage efficiency.

[0115] As a second aspect of the embodiments of the present application, an electronic device is provided, wherein: Figure 6 As shown, the electronic device includes:

[0116] One or more processors 101;

[0117] The memory 102 stores one or more computer programs. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the sharding storage method provided in the first aspect of the embodiment of the present application.

[0118] The electronic device may further include one or more I / O interfaces 103 connected between the processor 101 and the memory 102 and configured to implement information interaction between the processor 101 and the memory 102 .

[0119] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0120] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0121] As a third aspect of the embodiment of the present application, Figure 7As shown, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the shard storage method provided in the first aspect of the embodiment of the present application is implemented.

[0122] It can be understood by a person of ordinary skill in the art that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can implement the method of any of the above-mentioned embodiments when executed. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.

Claims

1. A shard storage method, characterized in that: The method comprises: Generate a globally unique identifier for a file to be stored; wherein the file to be stored includes a fragment file; For the shard file to be stored currently, according to the globally unique identifier, the storage paths of the current storage devices, the preset hash algorithm and the shard sequence identifier of the shard file to be stored currently, a shard file name and a target storage directory are generated for the shard file to be stored currently; Obtain encryption requirement information of the slice file to be stored currently; According to the segment file name and the encryption requirement information, the segment file currently to be stored is stored in the target storage directory.

2. The method according to claim 1, characterized in that The generating a fragment file name and a target storage directory for the fragment file to be stored currently according to the globally unique identifier, the current storage path of each storage device, the preset hash algorithm, and the fragment sequence identifier of the fragment file to be stored currently includes: Generate a fragment file name for the fragment file to be stored currently according to the globally unique identifier, the preset hash algorithm, and the fragment sequence identifier of the fragment file to be stored currently; Determine a target storage device from among the current storage devices according to the fragment sequence identifier of the fragment file to be stored currently; According to the storage path of the target storage device and the slice file name of the slice file to be stored currently, a target storage directory is generated for the slice file to be stored currently.

3. The method according to claim 2, characterized in that The step of generating a target storage directory for the slice file to be stored currently according to the storage path of the target storage device and the slice file name of the slice file to be stored currently includes: Generate a subdirectory according to the data located at a preset position in the name of the segment file of the segment file to be stored currently; wherein the subdirectory is a three-level directory; According to the storage path of the target storage device and the subdirectory, a target storage directory is generated for the slice file currently to be stored.

4. The method according to claim 1, characterized in that The obtaining encryption requirement information of the current shard file to be stored includes: Determining a storage encryption strategy for the file to be stored from a plurality of preset storage encryption strategies according to the attribute information of the file to be stored; The encryption requirement information of the current segmented file to be stored is determined at least according to the storage encryption policy of the file to be stored.

5. The method according to claim 4, characterized in that The plurality of preset storage encryption strategies include a small file full encryption strategy, a text file full encryption strategy, a large file first fragment encryption strategy, and a large file percentage encryption strategy; The step of determining the encryption storage strategy for the file to be stored from a plurality of preset encryption storage strategies according to the attribute information of the file to be stored includes: When it is determined according to the attribute information of the file to be stored that the size of the file to be stored does not exceed a preset threshold, the small file full encryption policy is determined as the encryption storage policy for the file to be stored; In the case that it is determined according to the attribute information of the file to be stored that the type of the file to be stored is a text file, the full encryption policy of the text file is determined as the encryption storage policy of the file to be stored; When it is determined based on the attribute information of the file to be stored that the size of the file to be stored exceeds the preset threshold and the type is not a text file, the large file first fragment encryption strategy and the large file percentage encryption strategy are displayed, an externally input policy indication message is received, and the encryption storage policy of the file to be stored indicated in the policy indication message is obtained.

6. The method according to claim 5, characterized in that The step of determining encryption requirement information of the current shard file to be stored at least according to the storage encryption policy of the file to be stored includes: When the storage encryption policy of the file to be stored is the small file full encryption policy or the text file full encryption policy, determining that the encryption requirement information of the current shard file to be stored is encryption required; In a case where the storage encryption policy of the file to be stored is the first shard encryption policy of the large file, determining encryption requirement information of the shard file to be stored according to the shard sequence identifier of the shard file to be stored currently; When the storage encryption policy of the file to be stored is the large file percentage encryption policy, the encryption requirement information of the slice file to be stored is determined according to the slice sequence identifier of the slice file to be stored, the total number of slice files in the file to be stored, and the percentage information indicated in the large file percentage encryption policy.

7. The method according to any one of claims 1 to 6, characterized in that The encryption requirement information of the current segment file to be stored is whether encryption is required or not required, and storing the current segment file to be stored in the target storage directory according to the segment file name and the encryption requirement information includes: In the case where the encryption requirement information of the currently to-be-stored segment file is that encryption is required, after encrypting the currently to-be-stored segment file according to a preset encryption algorithm, the encrypted segment file is stored in the target storage directory according to the segment file name; When the encryption requirement information of the segment file to be stored currently indicates that encryption is not required, the segment file to be stored currently is stored in the target storage directory according to the segment file name.

8. The method according to claim 7, characterized in that The preset encryption algorithm is any one of the following: XOR encryption algorithm, Data Encryption Standard DES algorithm, Triple Data Encryption Standard 3DES algorithm, Advanced Encryption Standard AES algorithm.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the sharding storage method according to any one of claims 1-7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the shard storage method according to any one of claims 1 to 7 is implemented.

Images