File-based data storage method, electronic equipment and computer readable medium
By adopting file-based data storage methods in personal/small storage and using preset storage mode policies (such as single copy, multiple copy, erasure code) for storage, the problem of waste and insufficient storage space in NAS technology is solved. Achieve more efficient and flexible storage reliability protection.
Patent Information
- Application Number
- CN202410962629.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
AI Technical Summary
The existing NAS technology has problems such as difficulty, high cost and inflexibility in realizing file reliability protection in the field of personal/small storage, especially the disk redundancy schemes that lead to waste and insufficient storage space.
Using file-based data storage method, the target storage strategy is determined through preset storage mode policies (single copy, multiple copy, single erasure code, multiple erasure code), and asynchronously write to the storage module to reduce user decision-making behavior. Improve reliability and store files as granularity to avoid wasting storage space.
It reduces the difficulty and cost of storage reliability protection, improves the flexibility of storage space utilization and data storage, and adapts to file shared storage with different reliability requirements.
Smart Images

Figure CN118915964A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of data storage, and in particular, to a file-based data storage method, electronic device, and computer-readable medium. Background Art
[0002] In the field of personal / small storage, Network Attached Storage (NAS) technology is generally used to provide file storage services. One of the solutions that NAS technology provides reliability for files is to organize disks into disk arrays (Redundant Arrays of Independent Disks, RAID), and then install file systems such as the fourth extended filesystem (ext4) based on RAID devices; another solution that NAS technology provides reliability for files is to use file systems with volume management functions. Such file systems have disk redundancy functions, such as the dynamic file system (Zettabyte File System, ZFS) that can build a disk pool (zpool), which can provide storage reliability similar to RAID.
[0003] However, both of the above solutions have disadvantages such as difficulty in implementation, high cost, and inflexibility. Summary of the invention
[0004] The present application aims to solve one of the technical problems in the related art to a certain extent. To this end, the present application provides a file-based data storage method, an electronic device and a computer-readable medium.
[0005] As a first aspect of the present application, a file-based data storage method is provided, the method comprising:
[0006] Obtain a target storage strategy for the file to be stored; wherein the target storage strategy is determined from a preset storage mode strategy, the preset storage mode strategy includes a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy, and an erasure code multiple storage strategy, and the target storage strategy is any one of the following or a combination thereof: one of the single copy storage strategy and the multiple copy storage strategy, one of the erasure code single storage strategy and the erasure code multiple storage strategy;
[0007] According to the target storage strategy, the file to be stored is stored in the target storage module;
[0008] The correspondence between the identifier of the file to be stored, the target storage strategy and the identifier of the target storage module is recorded.
[0009] Optionally, the preset storage mode strategy also includes a sharding strategy; when the target storage strategy includes the erasure code single storage strategy or the erasure code multiple storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy includes:
[0010] According to the sharding strategy, the file to be stored is divided into a plurality of shard data;
[0011] Obtaining the number of sub-data, the number of redundant data, and the redundancy algorithm configured by the erasure code single storage strategy or the erasure code multiple storage strategy included in the target storage strategy;
[0012] Divide each of the shard data of the sub-data amount into the same shard data group;
[0013] For any of the shard data groups, redundant data equal to the number of redundant data is calculated according to the redundancy algorithm and multiple shard data in the shard data group.
[0014] Optionally, when the target storage strategy includes the erasure code single storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy further includes:
[0015] For any of the shard data groups, a storage module matching the sum of the number of sub-data and the number of redundant data is determined from the current storage modules as a target storage module corresponding to the shard data group; wherein there are two shard data groups, and the target storage modules corresponding to the two shard data groups are not completely the same;
[0016] For any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in a target storage module corresponding to the shard data group.
[0017] Optionally, when the target storage strategy includes the erasure code multiple storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy further includes:
[0018] Obtaining the redundancy quantity of the erasure code configured by the erasure code multiple storage strategy;
[0019] Determine multiple groups of target storage modules from each current storage module according to the number of sub-data and the number of redundant data; wherein the total number of groups of the multiple groups of target storage modules matches the number of erasure code redundancies, and there is no intersection between any two groups of target storage modules;
[0020] For any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in each storage module of each group of target storage modules.
[0021] Optionally, the sharding strategy includes an equal sharding strategy or a quantity sharding strategy, wherein the equal sharding strategy is configured with an equal data size, and the quantity sharding strategy is configured with a quantity of sharded data.
[0022] Optionally, when the target storage strategy includes the single copy storage strategy or the multiple copy storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy includes:
[0023] Obtaining the number of replica copies configured by the single-copy storage strategy or the multi-copy storage strategy included in the target storage strategy;
[0024] According to a preset storage module selection strategy, a storage module matching the number of replica copies is determined from the current storage modules as a target storage module;
[0025] The files to be stored are stored in each of the target storage modules respectively.
[0026] Optionally, the method further comprises:
[0027] Receive a file access request sent by a client, and obtain the file identifier, offset, and access length to be accessed carried therein;
[0028] According to the identifier of the file to be accessed, query the correspondence between the identifier of the file recorded locally, the storage strategy, and the identifier of the storage module, and determine the target storage strategy of the file to be accessed and the storage module to be read;
[0029] Determine the file to be accessed according to the target storage strategy of the file to be accessed and the storage module to be read;
[0030] The data corresponding to the offset and the access length in the file to be accessed is returned to the client.
[0031] Optionally, obtaining a target storage strategy for the file to be stored includes:
[0032] Acquire attribute information of the file to be stored;
[0033] Based on the attribute information and pre-configured storage mode policy usage rules, the target storage strategy for the file to be stored is determined; wherein the storage mode policy usage rules include a correspondence between attribute information conditions and storage mode policies, and the attribute information conditions correspond to any of the following attribute information or a combination thereof: file size information, file type information, file subtype information, file user identification, file copyright user information, file device source information, and file business source information.
[0034] As a second aspect of the present application, an electronic device is provided, wherein the electronic device includes:
[0035] one or more processors;
[0036] A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the file-based data storage method described in the first aspect of the present application.
[0037] As a third aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the file-based data storage method described in the first aspect of the present application is implemented.
[0038] In the file-based data storage method provided in the embodiment of the present application, a target storage strategy for the file to be stored is obtained; wherein the target storage strategy is determined from a preset storage mode strategy, the preset storage mode strategy includes a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy and an erasure code multiple storage strategy, and the target storage strategy is any one of the following or a combination thereof: one of the single copy storage strategy and the multiple copy storage strategy, one of the erasure code single storage strategy and the erasure code multiple storage strategy; according to the target storage strategy, the file to be stored is stored in the target storage module; the corresponding relationship between the identifier of the file to be stored, the target storage strategy and the identifier of the target storage module is recorded. There is no need to form a RAID of disks, and storage is no longer performed at the granularity of disks, but at the granularity of files, which reduces user decision-making behavior and improves the usability of the storage module; by presetting multiple storage mode strategies, the file to be stored is asynchronously written to the storage module, which improves reliability, and files with different reliability requirements share the storage module, which avoids the problem of storage space waste and insufficient storage space, and improves the utilization of storage space and the flexibility of data storage. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The present application will be further described below in conjunction with the accompanying drawings:
[0040] Figure 1 This is a flowchart of an implementation method of a file-based data storage method provided in an embodiment of the present application;
[0041] Figure 2 It is a flowchart of an implementation method of storing a file to be stored in a target storage module provided by an embodiment of the present application;
[0042] Figure 3 It is a flowchart of another implementation method of storing a file to be stored in a target storage module provided by an embodiment of the present application;
[0043] Figure 4 It is a flowchart of another implementation method of storing a file to be stored in a target storage module provided by an embodiment of the present application;
[0044] Figure 5 It is a flowchart of another implementation method of storing the file to be stored in the target storage module provided by the embodiment of the present application;
[0045] Figure 6 is a flowchart of another implementation of the file-based data storage method provided in an embodiment of the present application;
[0046] Figure 7 This is a flowchart of an implementation method of the inspection and repair process provided in an embodiment of the present application;
[0047] Figure 8 It is a flowchart of another implementation method of the file-based data storage method provided in an embodiment of the present application;
[0048] Fig. 9 This is a flowchart of an implementation method of a process for modifying a target storage policy provided in an embodiment of the present application;
[0049] Fig.10 This is a flowchart of an implementation method of obtaining a target storage strategy for files to be stored provided in an embodiment of the present application;
[0050] Fig.11 It is a module schematic diagram of an implementation of a data storage system provided in an embodiment of the present application;
[0051] Fig.12 It is a module schematic diagram of an implementation mode of an electronic device provided in an embodiment of the present application;
[0052] Fig.13 It is a schematic diagram of a computer-readable medium provided in an embodiment of the present application.
[0053] Description of Reference Numerals
[0054] 101: Processor 102: Memory
[0055] 103: I / O interface 104: bus DETAILED DESCRIPTION
[0056] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments based on the embodiments are intended to be used to explain the present application and cannot be understood as limiting the present application.
[0057] References to "one embodiment" or "an example" or "an example" in this specification mean that a particular feature, structure, or characteristic described in conjunction with the embodiment itself may be included in at least one embodiment disclosed in the present application. The appearance of the phrase "in one embodiment" in various places in the specification does not necessarily refer to the same embodiment.
[0058] At present, the NAS technology has the disadvantages of being difficult to implement, costly, and inflexible in providing reliability for files. For example, the RAID level of the disk must be determined when the device is unpacked and initialized, but RAID is a professional technology that is difficult to understand, plan, and implement. For another example, in a personal / small storage module, most of the data that takes up more space is movies and videos, which do not require additional reliability protection due to the low acquisition cost. However, the data that really needs high reliability protection takes up less space. The solution that provides reliability protection in units of disks is generally in units of T, which will cause a certain amount of storage space waste. For example, in the solution of combining two 1T disks into RAID1, the amount of data that the user actually needs to protect is likely to be much less than 1T. For another example, in a personal / small storage module with four 1T disks, two 1T disks are combined into RAID1 and two 1T disks are combined into RAID0. However, after RAID0 is used up, it is difficult to use the space of RAID1 without modifying the business system.
[0059] In view of this, the inventor of the present application proposes that, in a personal / small storage module, reliability protection is implemented based on files, which can effectively reduce the difficulty of reliability protection, reduce storage space costs and storage flexibility.
[0060] As a first aspect of an embodiment of the present application, a file-based data storage method is provided, such as Figure 1 As shown, the method may include:
[0061] In step S110, a target storage strategy for the file to be stored is obtained; wherein the target storage strategy is determined from a preset storage mode strategy, and the preset storage mode strategy includes a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy, and an erasure code multiple storage strategy, and the target storage strategy is any one of the following or a combination thereof: one of the single copy storage strategy and the multiple copy storage strategy, and one of the erasure code single storage strategy and the erasure code multiple storage strategy;
[0062] In step S120, according to the target storage strategy, the file to be stored is stored in the target storage module;
[0063] In step S130, the correspondence between the identifier of the file to be stored, the target storage policy and the identifier of the target storage module is recorded.
[0064] The file-based data storage method provided in the embodiment of the present application can be applied to a data storage system, which may include multiple storage modules. A storage module can be a block device with a built file system, and a block device can be a disk or a disk partition. It should be noted that a storage module can also be a virtual storage module of a network attached storage (NAS) or a virtual storage module implemented based on a cloud abstract file system.
[0065] The file to be stored may be local data of the data storage system (e.g., data in a mobile hard disk), or data that is remotely transmitted and requested to be uploaded by the client. When the file to be stored is data that is remotely transmitted and requested to be uploaded by the client, after storing the file to be stored in the target storage module according to the target storage policy, the data storage system may also return the identifier of the file to be stored to the client to notify the client that the storage of the corresponding file has been completed.
[0066] Among them, the preset storage mode strategy may include a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy, and an erasure code multiple storage strategy. In the embodiment of the present application, there is no specific limitation on how to obtain the target storage strategy for the file to be stored. It can be determined by the client from the preset storage mode strategy, or it can be determined by the data storage system from the preset storage mode strategy according to a certain rule.
[0067] The target storage strategy can be one of a single copy storage strategy and a multiple copy storage strategy, or one of an erasure code single storage strategy and an erasure code multiple storage strategy, or one of a single copy storage strategy and a multiple copy storage strategy, or one of a single copy storage strategy and a multiple copy storage strategy. Both the single copy storage strategy and the multiple copy storage strategy belong to the replica storage strategy, and both the erasure code single storage strategy and the erasure code multiple storage strategy belong to the erasure code storage strategy, that is, in the embodiment of the present application, both the replica storage strategy and the erasure code storage strategy can be used as the target storage strategy at the same time.
[0068] In the embodiment of the present application, the preset storage mode strategy can be customized by the user or can be a factory definition of the storage module, and the embodiment of the present application does not specifically limit this.
[0069] In the file-based data storage method provided in the embodiment of the present application, a target storage strategy for the file to be stored is obtained; wherein the target storage strategy is determined from a preset storage mode strategy, the preset storage mode strategy includes a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy and an erasure code multiple storage strategy, and the target storage strategy is any one of the following or a combination thereof: one of the single copy storage strategy and the multiple copy storage strategy, one of the erasure code single storage strategy and the erasure code multiple storage strategy; according to the target storage strategy, the file to be stored is stored in the target storage module; the corresponding relationship between the identifier of the file to be stored, the target storage strategy and the identifier of the target storage module is recorded. There is no need to form a RAID of disks, and storage is no longer performed at the granularity of disks, but at the granularity of files, which reduces user decision-making behavior and improves the usability of the storage module; by presetting multiple storage mode strategies, the file to be stored is asynchronously written to the storage module, which improves reliability, and files with different reliability requirements share the storage module, which avoids the problem of storage space waste and insufficient storage space, and improves the utilization of storage space and the flexibility of data storage.
[0070] The erasure code algorithm is also a redundancy algorithm, which refers to generating N redundant data based on M sub-data. In this way, when some of the (M+N) data are damaged or lost, they can be restored based on the remaining data. In the embodiment of the present application, the preset storage mode strategy may also include a sharding strategy. When the erasure code single storage strategy or the erasure code multiple storage strategy is used as the target storage strategy, the storage file needs to be sharded according to the sharding strategy.
[0071] Correspondingly, in some embodiments, the preset storage mode strategy also includes a sharding strategy; when the target storage strategy includes the erasure code single storage strategy or the erasure code multiple storage strategy, in the step of storing the file to be stored in the target storage module according to the target storage strategy (i.e., step S120), as Figure 2 As shown, it may include:
[0072] In step S121, the file to be stored is divided into a plurality of shard data according to the shard strategy;
[0073] In step S122, the number of sub-data, the number of redundant data, and the redundancy algorithm configured by the erasure code single storage strategy or the erasure code multiple storage strategy included in the target storage strategy are obtained;
[0074] In step S123, each of the shard data of the sub-data amount is divided into the same shard data group;
[0075] In step S124, for any of the shard data groups, redundant data equal to the number of redundant data is calculated according to the redundancy algorithm and the plurality of shard data in the shard data group.
[0076] Among them, M and N represent the number of sub-data and the number of redundant data involved in the redundancy algorithm, respectively. Whether it is the single-time storage strategy of erasure code or the multiple-time storage strategy of erasure code, M, N and the redundancy algorithm need to be configured, and the corresponding N redundant data need to be generated for each M sub-data according to the redundancy algorithm. The difference is that when the single-time storage strategy of erasure code is adopted, it is only necessary to store each M sub-data and the corresponding N redundant data in one group of storage modules; while when the multiple-time storage strategy of erasure code is adopted, each M sub-data and the corresponding N redundant data need to be stored in multiple groups of storage modules at the same time, which is equivalent to storing each M sub-data and its corresponding N redundant data multiple times.
[0077] In addition, it should be noted that in the embodiment of the present application, the total number of shard data can be consistent with the amount of sub-data, that is, the file to be stored can be divided into M shard data, that is, M sub-data. Correspondingly, at this time, there is actually only one shard data group.
[0078] In some embodiments, when the target storage strategy includes the erasure code single storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy (ie, step S120) is as follows: Figure 3 As shown, it may also include:
[0079] In step S125a, for any of the shard data groups, a storage module that matches the sum of the number of sub-data and the number of redundant data is determined from the current storage modules as a target storage module corresponding to the shard data group; wherein there are two shard data groups, and the target storage modules corresponding to the two shard data groups are not completely the same;
[0080] In step S126a, for any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in a target storage module corresponding to the shard data group.
[0081] Among them, each shard data group includes M shard data, and 1 shard data is used as 1 sub-data. For each shard data group, that is, for each M shard data and its corresponding N redundant data, first determine (M+N) storage modules from the current storage modules of the system as the target storage modules corresponding to the shard data group.
[0082] Furthermore, when the total number of current storage modules of the system is greater than (M+N), there are two shard data groups, and the corresponding target storage modules are not completely the same. Assuming (M+N) is P and the total number of current storage modules of the system is (P+1), that is, when the identifiers of the target storage modules corresponding to shard data group x are 1, 2, 3, ..., P respectively, the identifiers of the target storage modules corresponding to shard data group y may be 2, 3, ..., (P+1) or 1, 3, ..., (P+1) or 1, 2, 4, ..., (P+1) and so on.
[0083] By storing M different shard data and their corresponding N redundant data in different target storage modules, the amount of damaged data caused by storage module damage can be effectively reduced, thereby effectively improving the recoverability and security of data storage.
[0084] As described above, when the erasure code multiple storage strategy is adopted, it is equivalent to storing each M sub-data and its corresponding N redundant data multiple times. Accordingly, in some embodiments, when the target storage strategy includes the erasure code multiple storage strategy, the step of storing the to-be-stored file in the target storage module according to the target storage strategy (i.e., step S120) is as follows: Figure 4 As shown, it may also include:
[0085] In step S125b, the erasure code redundancy quantity configured by the erasure code multiple storage strategy is obtained;
[0086] In step S126b, according to the number of sub-data and the number of redundant data, multiple groups of target storage modules are determined from the current storage modules; wherein the total number of groups of the multiple groups of target storage modules matches the number of erasure code redundancies, and there is no intersection between any two groups of target storage modules;
[0087] In step S127b, for any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in each storage module of each group of target storage modules.
[0088] Among them, the embodiment of the present application does not make any specific limitation on the number of erasure code redundancies configured for the erasure code multiple storage strategy (that is, how many times each M sub-data and its corresponding N redundant data are stored). As a preferred implementation method, in order to save storage space and improve storage efficiency, the number of erasure code redundancies can be 2.
[0089] When the erasure code multiple storage strategy is adopted as the target storage strategy, Q represents the erasure code redundancy number, and Q groups of target storage modules are determined from the current storage modules of the system. The total number of storage modules in each group of target storage modules is (M+N), and there is no intersection between any two groups of target storage modules in the Q groups of target storage modules.
[0090] For example, M is 3, N is 2, (M+N) is 5, the first group of target storage modules includes target storage module 1, target storage module 2, target storage module 3, target storage module 4, target storage module 5, the second group of target storage modules includes target storage module 6, target storage module 7, target storage module 8, target storage module 9, target storage module 10... and so on. For any shard data group, not only the three shard data and the corresponding two redundant data are stored in the target storage module 1, target storage module 2, target storage module 3, target storage module 4, and target storage module 5 in a one-to-one correspondence, but also the three shard data and the corresponding two redundant data are stored in the target storage module 6, target storage module 7, target storage module 8, target storage module 9, and target storage module 10 in a one-to-one correspondence... and so on.
[0091] In an embodiment of the present application, as a preferred implementation, in order to save storage space and improve storage efficiency, when the total number of current storage modules in the system is large (for example, exceeds a certain threshold), the erasure code multiple storage strategy is adopted as the target storage strategy.
[0092] As described above, when adopting the erasure code storage strategy, the file to be stored must be sharded first, and the preset sharding strategy can be sharding according to a fixed or variable equal data size, or sharding according to a fixed or variable number of sharded data. Accordingly, in some embodiments, the sharding strategy includes an equal sharding strategy or a number sharding strategy, the equal sharding strategy is configured with an equal data size, and the number sharding strategy is configured with a number of sharded data.
[0093] Among them, the embodiments of the present application do not make any specific limitations on the fixed size of equally divided data. For example, the fixed size of equally divided data can be 1M, 2M, 3M, etc. The embodiments of the present application do not make any specific limitations on the variable size of equally divided data. For example, the variable size of equally divided data can be determined based on the size of the file to be stored and the total number of current storage modules of the system. The embodiments of the present application do not make any specific limitations on the fixed number of sharded data. For example, the fixed number of sharded data can be 2, 3, 4, etc. The embodiments of the present application do not make any specific limitations on the variable number of sharded data. For example, the variable number of sharded data can be determined based on the total number of current storage modules of the system.
[0094] When sharding a storage file according to the equal-sharding strategy, when the size of the last shard data is smaller than the equal-shard data size, it can be filled with blanks. That is to say, at this time, the sharding obtains multiple shard data with a data length equal to the equal-shard data size and one shard data with an actual data length smaller than the equal-shard data size.
[0095] In an embodiment of the present application, in addition to the above-mentioned replica storage strategy and erasure code storage strategy, as an optional implementation method, the file to be stored can be divided into multiple shard data according to a preset shard strategy, and then each preset number of shard data is taken as a data block unit, and each data block unit is stored in each storage module respectively.
[0096] When a single copy storage strategy or a multiple copy storage strategy is adopted as the target storage strategy, it refers to whether the file to be stored is stored once or multiple times. Accordingly, in some embodiments, when the target storage strategy includes the single copy storage strategy or the multiple copy storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy (i.e., step S120) is as follows. Figure 5 As shown, it may also include:
[0097] In step S121', the number of replica copies configured by the single-copy storage strategy or the multi-copy storage strategy included in the target storage strategy is obtained;
[0098] In step S122', according to a preset storage module selection strategy, a storage module matching the number of replica copies is determined from the current storage modules as a target storage module;
[0099] In step S123', the files to be stored are stored in each of the target storage modules respectively.
[0100] Among them, whether it is a single copy storage strategy or a multi-copy storage strategy, the number of copies needs to be configured. The preset storage module selection strategy can be a space balance strategy or a most idle strategy. The space balance strategy refers to selecting the storage module with the largest remaining storage space, and the most idle strategy refers to selecting the storage module with the most idle current input / output (I / O).
[0101] For example, a multi-copy storage strategy is used as the target storage strategy. The number of copies configured for the multi-copy storage strategy is 2. The total number of current storage modules in the system is 5. Then, the two storage modules with the largest remaining storage space are selected from the five storage modules as the target storage modules, and the files to be stored are stored in these two target storage modules respectively.
[0102] In an embodiment of the present application, a file to be stored can also be completely stored in a storage module in an integral form or in a fragmented form. Accordingly, in some embodiments, in the step of storing the file to be stored in each of the target storage modules respectively, it also includes: according to the fragmentation strategy, dividing the file to be stored into multiple fragment data, and storing the multiple fragment data in each of the target storage modules.
[0103] In addition to requesting the data storage system to store files, the client can also request the data storage system to access files based on the file identifier. Figure 6 As shown, the method may also include:
[0104] In step S210, a file access request sent by a client is received, and the identifier, offset and access length of the file to be accessed carried therein are obtained;
[0105] In step S220, according to the identifier of the file to be accessed, the correspondence between the identifier of the file recorded locally, the storage strategy, and the identifier of the storage module is queried to determine the target storage strategy of the file to be accessed and the storage module to be read;
[0106] In step S230, the file to be accessed is determined according to the target storage policy of the file to be accessed and the storage module to be read;
[0107] In step S240, the data corresponding to the offset and the access length in the file to be accessed is returned to the client.
[0108] In an embodiment of the present application, a file access request may carry an identification (Identity, ID) of the file to be accessed, an offset (offset) and an access length (len). When the data storage system receives the file access request sent by the client, it executes data reading, namely, steps S210 to S240.
[0109] It is understandable that when the target storage strategy of the file to be accessed is only a single copy storage strategy, there is only one storage module to be read. When the target storage strategy of the file to be accessed includes a multiple copy storage strategy or an erasure code single storage strategy or an erasure code multiple storage strategy, there are multiple storage modules to be read, and if the data fails to be read from one or part of the storage modules to be read, data recovery can be performed. For example, when the target storage strategy includes a multiple copy storage strategy, there are multiple storage modules to be read, and the file to be accessed can be read from another storage module to be read except the storage module to be read that failed to be read, and the corresponding file to be accessed in the storage module to be read that failed to be read is replaced. For another example, when the target storage strategy includes an erasure code single storage strategy or an erasure code multiple storage strategy, M sub-data and M redundant data that are interrelated can be read, and the sub-data or redundant data that failed to be read can be recovered according to the redundancy algorithm.
[0110] Of course, the embodiments of the present application are not limited to this. The reading failure may be caused by various reasons such as damage to the storage module itself or damage to the file system of the storage module. Therefore, the inspection and repair process can be triggered when the reading fails, and the storage module to be read that has failed to be read can be inspected and repaired with the help of the inspection and repair process.
[0111] like Figure 7As shown, the inspection and repair process may include the following steps: a. Check the storage module. If the storage module is no longer online and is still not online after restart, jump to f, otherwise jump to b; b. Check the file system of the storage module. If the file system of the storage module is damaged, repair the file system and execute c; c. For each data in the repaired file system, according to the correspondence between the local recorded file identifier, storage policy, and storage module identifier, all data related to the current data are obtained by searching or traversing to verify the current data to obtain the verification result of the current data. If the verification fails, add it to the repair failure list. If the repair failure list is empty, jump to f; if the repair failure list is not empty, jump to d; d. Determine the repairable data in the repair failure list to obtain a repairable list and jump to e; e. For each repairable data in the repairable list, according to the correspondence between the local recorded file identifier, storage policy, and storage module identifier, perform repair and jump to f; f. End the process.
[0112] In addition, the above check and repair process can be triggered not only when reading the to-be-accessed file fails, but also according to a preset period to further improve the reliability and security of data storage. Figure 8 As shown, the method may also include:
[0113] In step S310, according to a preset period, based on the correspondence between the identifier of the locally recorded file, the storage policy, and the identifier of the storage module, the data in each storage module is verified and the data that fails the verification is repaired.
[0114] In the embodiment of the present application, there is no special limitation on the preset period, for example, the preset period can be set to 5 days, 1 week, 1 month, etc. Whenever a new period is reached, the data storage system performs a file consistency and integrity check, specifically, starts to traverse all the data in each storage module, obtains data belonging to the same file as the current data according to the correspondence between the file identifier, storage policy, and storage module identifier recorded locally, and verifies the correctness of the current data.
[0115] In addition to accessing files, the client can also request the data storage system to delete data, modify the current storage mode policy, etc. When receiving a request to delete data from the client, the data storage system can execute a garbage cleaning process to delete the relevant data. The garbage cleaning process may include: traversing each garbage cleaning task, performing a reverse reference check on the data to be cleaned by the current garbage cleaning task, and if the data still needs to be associated with other data (other files, sub-data, or redundant data) belonging to the same file, skipping the current garbage cleaning task, otherwise deleting the data.
[0116] When receiving a request from a client to modify the storage policy of a file, the data storage system can execute the storage policy modification process. Fig. 9 As shown, the storage strategy modification process may include the following steps: a. Determine the modification type. If the modification type is a redundancy increase type, jump to b1; if the modification type is a redundancy reduction type, jump to b2; if the modification type is a redundancy conversion type, jump to b3; b1. Add a file copy and modify the correspondence between the identifier of the locally recorded file, the storage strategy, and the identifier of the storage module;
[0117] b2. Reduce file copies and modify the correspondence between the local recorded file identification, storage strategy, and storage module identification; b3. Add conversion copies according to the new redundancy algorithm carried in the request, delete the original copies corresponding to the old redundancy algorithm, and modify the local recorded file identification, storage strategy, and storage module identification.
[0118] As described above, the client may determine the target storage strategy for the file to be stored from the preset storage mode strategy, or the data storage system may determine the target storage strategy for the file to be stored from the preset storage mode strategy according to a certain rule. Accordingly, in some embodiments, the target storage strategy for the file to be stored (i.e., what is involved in step S110) is obtained, such as Fig.10 As shown, it may include:
[0119] In step S111, the attribute information of the file to be stored is obtained;
[0120] In step S112, the target storage strategy for the file to be stored is determined based on the attribute information and the pre-configured storage mode policy usage rules; wherein the storage mode policy usage rules include a correspondence between attribute information conditions and storage mode policies, and the attribute information conditions correspond to any of the following attribute information or a combination thereof: file size information, file type information, file subtype information, file user identification, file copyright user information, file device source information, and file business source information.
[0121] The attribute information of the file may include general attribute information of the file, such as file size information, file type information, file subtype information, file user identification, etc., and may also include extensible attributes of the file, such as file copyright user information, file device source information, file service source information, etc. Specifically, the file size information may be in bytes, the file type information may include pictures, videos, audios, documents, etc., the file subtype information may be the result of further dividing the file type information, such as JPG format pictures, animated pictures, etc., the file user identification is the identification of the user to which the file currently belongs, the file copyright user information may be the identification of the user to which the file belongs when it is created, the file device source information may include the types and identifications of all devices requesting to store the file, and the file service source information may include the identification of the service that generates the file, such as the identification of a group session.
[0122] Any of the above-mentioned attribute information or a combination thereof can be used to configure storage mode policy usage rules. Accordingly, one or more storage mode policy usage rules can be pre-configured. Furthermore, when configuring multiple storage mode policy usage rules, the priority of each storage mode policy usage rule can also be configured synchronously. The priority can be represented by text, numbers, letters, etc. As an implementation method that is most easily recognizable, numbers can be used to represent the priority. For example, the priority can be represented from large to small in the order of numbers. The lowest priority can be set to 1.
[0123] In an embodiment of the present application, by matching the attribute information of the file to be stored with the attribute information conditions in the pre-configured storage mode policy usage rules, it is determined whether the file to be stored matches the storage mode policy usage rules, so that the storage mode policy in the matched storage mode policy usage rules is used as the target storage policy for the file to be stored. When there are multiple pre-configured storage mode policy usage rules, the attribute information of the file to be stored can be matched one by one in the order of priority from large to small for the multiple storage mode policy usage rules. When there are multiple storage mode policy usage rules with the same priority, the attribute information of the file to be stored can be matched one by one in the order of configuration or recording of the multiple storage mode policy usage rules.
[0124] For example, the following Table 1 records 4 storage mode policy usage rules. The priority of the first storage mode policy usage rule is 11, where the attribute information condition includes the file business source information "group chat-100", and the storage mode policy used is the multi-copy storage strategy; the priority of the second storage mode policy usage rule is 10, where the attribute information condition includes the file size information "<10M", and the storage mode strategy used is the multi-copy storage strategy; the priority of the third storage mode policy usage rule is 10, where the attribute information condition includes the file size information ">2G", and the storage mode strategy used is the erasure code single storage strategy; the priority of the fourth storage mode policy usage rule is 1, where the attribute information conditions include the file size information, file type information, file subtype information, file user identification, file copyright user information, file device source information, and file business source information, which can all be any values, and the storage mode strategy used is the single copy storage strategy. When the matching result is that the attribute information of the file to be stored cannot match the attribute information condition that the file business source information is "Group Chat-100", since the priority of the second storage mode policy usage rule and the third storage mode policy usage rule are both 10 but the recording order of the second storage mode policy usage rule is first, the attribute information of the file to be stored is first matched with the attribute information condition of the second storage mode policy usage rule.
[0125] Table 1
[0126]
[0127] In addition, regarding the reliability of the file to be stored, when the file to be stored is local data of the data storage system, there is no need to perform reliability verification. When the file to be stored is data transmitted remotely from the client, reliability verification of the file to be stored can be considered. Accordingly, in some embodiments, the method may also include: according to the file upload request sent by the client, reliability verification of the file to be stored sent by the client. Accordingly, the acquisition of the target storage policy of the file to be stored (i.e., involved in step S110) may include: when the reliability verification passes, acquiring the target storage policy of the file to be stored.
[0128] The client may send a file upload request and a file to be stored to the data storage system at the same time, or may first send a file upload request to the data storage system and then send the file to be stored after the data storage system confirms the resource and returns a response to it.
[0129] As an optional implementation, verifying the reliability of the file to be stored sent by the client may include: comparing the file size carried in the file upload request with the size of the file to be stored; and / or calculating the summary value of the file to be stored, and comparing the calculated summary value with the summary value carried in the file upload request.
[0130] The client may carry the file size (size) of the file to be stored and / or the digest value (Md5 value or Sha256 value) of the file to be stored in the file upload request sent to the data storage system. After the data storage system receives the file to be stored, if the file size (size) has been received before, it compares it with the size of the file to be stored; if the digest value (Md5 value or Sha256 value) has been received before, it compares it with the calculated digest value of the file to be stored.
[0131] It is understandable that when the received file size does not match the size of the file to be stored, and when the received digest value (Md5 value or Sha256 value) does not match the calculated digest value of the file to be stored, it can be considered that the reliability verification has failed.
[0132] like Fig.11As shown, it is a module schematic diagram of an implementation of the data storage system provided in the embodiment of the present application. It can be seen that the data storage system may include a file management module, a data receiving module, a data sending module, a data reliability module and a plurality of data storage modules. The data user, i.e., the client, may request access to the data storage system, such as an application (Application, APP), a personal computer (personal computer, PC), etc. The file management module is used to manage the preset storage mode strategy, the pre-configured storage mode strategy usage rules, the storage module selection strategy, the correspondence between the identification of the record file, the storage strategy, and the identification of the storage module, etc. The data receiving module is used to receive the file to be stored sent by the client and forward it to the data reliability module, and the data sending module is used to receive the data to be accessed sent by the data reliability module and forward it to the client. Both the data receiving module and the data sending module can support different protocols, such as HyperText Transfer Protocol (HTTP) and Protocol Buffers (PB). The data reliability module is used to save the file to be stored to the storage module according to the preset storage module selection strategy and the target storage strategy of the file to be stored, or read the data requested by the client from the storage module to be read. Data storage module, a data storage system has at least one data storage module, a data storage module can be a block device with a built file system, a block device can be a disk or a disk partition, or a storage module can also be a virtual storage module of a network attached storage (NAS), or a virtual storage module implemented based on a cloud abstract file system.
[0133] As a second aspect of the embodiments of the present application, an electronic device is provided, wherein: Fig.12 As shown, the electronic device includes:
[0134] One or more processors 101;
[0135] The memory 102 stores one or more computer programs. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the file-based data storage method provided in the first aspect of the embodiment of the present application.
[0136] The electronic device may further include one or more I / O interfaces 103 connected between the processor 101 and the memory 102 and configured to implement information interaction between the processor 101 and the memory 102 .
[0137] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the processor and the memory, including but not limited to a data bus (Bus), etc.
[0138] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.
[0139] As a third aspect of the embodiment of the present application, Fig.13 As shown, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the file-based data storage method provided in the first aspect of the embodiment of the present application is implemented.
[0140] It can be understood by a person of ordinary skill in the art that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can implement the method of any of the above-mentioned embodiments when executed. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0141] The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.
Claims
1. A file-based data storage method, characterized in that: The method comprises: Obtain a target storage strategy for the file to be stored; wherein the target storage strategy is determined from a preset storage mode strategy, the preset storage mode strategy includes a single copy storage strategy, a multiple copy storage strategy, an erasure code single storage strategy, and an erasure code multiple storage strategy, and the target storage strategy is any one of the following or a combination thereof: one of the single copy storage strategy and the multiple copy storage strategy, one of the erasure code single storage strategy and the erasure code multiple storage strategy; According to the target storage strategy, the file to be stored is stored in the target storage module; The correspondence between the identifier of the file to be stored, the target storage strategy and the identifier of the target storage module is recorded.
2. The method according to claim 1, characterized in that The preset storage mode strategy also includes a sharding strategy; when the target storage strategy includes the erasure code single storage strategy or the erasure code multiple storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy includes: According to the sharding strategy, the file to be stored is divided into a plurality of shard data; Obtaining the number of sub-data, the number of redundant data, and the redundancy algorithm configured by the erasure code single storage strategy or the erasure code multiple storage strategy included in the target storage strategy; Divide each of the shard data of the sub-data amount into the same shard data group; For any of the shard data groups, redundant data equal to the number of redundant data is calculated according to the redundancy algorithm and multiple shard data in the shard data group.
3. The method according to claim 2, characterized in that When the target storage strategy includes the erasure code single storage strategy, the step of storing the to-be-stored file in the target storage module according to the target storage strategy further includes: For any of the shard data groups, a storage module matching the sum of the number of sub-data and the number of redundant data is determined from the current storage modules as a target storage module corresponding to the shard data group; wherein there are two shard data groups, and the target storage modules corresponding to the two shard data groups are not completely the same; For any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in a target storage module corresponding to the shard data group.
4. The method according to claim 2, characterized in that: When the target storage strategy includes the erasure code multiple storage strategy, the step of storing the to-be-stored file in the target storage module according to the target storage strategy further includes: Obtaining the redundancy quantity of the erasure code configured by the erasure code multiple storage strategy; Determine multiple groups of target storage modules from the current storage modules according to the number of sub-data and the number of redundant data; wherein the total number of the multiple groups of target storage modules matches the number of erasure code redundancies, and there is no intersection between any two groups of target storage modules; For any of the shard data groups, multiple shard data and corresponding redundant data in the shard data group are stored one-to-one in each storage module of each group of target storage modules.
5. The method according to claim 2, characterized in that: The sharding strategy includes an equal sharding strategy or a quantity sharding strategy. The equal sharding strategy is configured with an equal data size, and the quantity sharding strategy is configured with a quantity of sharded data.
6. The method according to claim 1, characterized in that When the target storage strategy includes the single copy storage strategy or the multiple copy storage strategy, the step of storing the file to be stored in the target storage module according to the target storage strategy includes: Obtaining the number of replica copies configured by the single-copy storage strategy or the multi-copy storage strategy included in the target storage strategy; According to a preset storage module selection strategy, a storage module matching the number of replica copies is determined from the current storage modules as a target storage module; The files to be stored are stored in each of the target storage modules respectively.
7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Receive a file access request sent by a client, and obtain the file identifier, offset, and access length to be accessed carried therein; According to the identifier of the file to be accessed, query the correspondence between the identifier of the file recorded locally, the storage strategy, and the identifier of the storage module, and determine the target storage strategy of the file to be accessed and the storage module to be read; Determine the file to be accessed according to the target storage strategy of the file to be accessed and the storage module to be read; The data corresponding to the offset and the access length in the file to be accessed is returned to the client.
8. The method according to any one of claims 1 to 6, characterized in that The step of obtaining a target storage strategy for files to be stored includes: Acquire attribute information of the file to be stored; Based on the attribute information and pre-configured storage mode policy usage rules, the target storage strategy for the file to be stored is determined; wherein the storage mode policy usage rules include a correspondence between attribute information conditions and storage mode policies, and the attribute information conditions correspond to any of the following attribute information or a combination thereof: file size information, file type information, file subtype information, file user identification, file copyright user information, file device source information, and file business source information.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the file-based data storage method according to any one of claims 1-8.
10. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file-based data storage method according to any one of claims 1 to 8 is implemented.
Citation Information
Cited By
Data processing method and device, medium and program product
CN120872264A
