Method and device for storing upper layer application data of distributed storage as erasure code

Abstract

The present invention discloses a method for storing upper-layer application data of distributed storage as erasure codes, comprising: when the service is initialized, reading a preset data file storage configuration name; determining whether the data file storage configuration name is a configuration name within a merge strategy; if so, writing a target data file within a preset time range into a pre-created temporary subdirectory according to time stratification, and converting the format of the target data file according to a preset first erasure code format; if not, converting the format of stored data files and data files to be stored in combination with the pre-configured second erasure code formats corresponding to each level of storage and the preset storage configuration priorities at each level. The method for storing upper-layer application data of distributed storage as erasure codes provided by the present invention reduces storage costs and management complexity. The present invention also discloses a device, equipment and storage medium for storing upper-layer application data of distributed storage as erasure codes, which have corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a method, device, equipment and computer-readable storage medium for storing upper-layer application data of distributed storage as erasure codes. Background Art

[0002] The Erasure Coding (EC) strategy is a coding technology that provides the same fault tolerance as the default replication mechanism of HDFS (Hadoop Distributed File System, which refers to a distributed file system designed to run on general-purpose hardware). It occupies less storage space but consumes more resources when reading. It is suitable for use when there is surplus central processing unit (CPU) resources and high storage costs.

[0003] When the query frequency of some data in the HBase cluster is not high and the storage space is insufficient, the data files need to be stored as erasure codes. Currently, only the HDFS directory where some HBase data is located can be manually set to the erasure code format using commands. However, when the entire database, table level, column family (CF) level, or different time-layer levels of the same column family need to be replaced with the erasure code format, the operation is complicated and inconvenient to manage and convert existing data.

[0004] In summary, how to effectively solve the problems of manually setting the erasure code format, which is complicated, inconvenient to manage, and conversion of existing data, is a problem that technicians in this field urgently need to solve. Summary of the invention

[0005] The purpose of the present invention is to provide a method for storing upper-layer application data of distributed storage as erasure codes. The method flexibly and conveniently specifies part or all of the data as erasure code format by changing the hbase configuration, thereby reducing storage costs and management complexity. Another purpose of the present invention is to provide an apparatus, device and computer-readable storage medium for storing upper-layer application data of distributed storage as erasure codes.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A method for storing upper-layer application data of distributed storage as erasure codes, comprising:

[0008] When the service is initialized, read the preset data file storage configuration name;

[0009] Determine whether the data file storage configuration name is a configuration name within the merge strategy;

[0010] If so, write the target data file of the preset time range into a pre-created temporary subdirectory according to the time layer, and convert the format of the target data file according to the preset first erasure code format; wherein the temporary subdirectory is a subdirectory created under the temporary directory of the column family;

[0011] If not, the formats of the stored data files and the data files to be stored are converted in combination with the second erasure code formats corresponding to the pre-configured storage levels and the preset storage configuration priorities at each level.

[0012] In a specific implementation manner of the present invention, after the format conversion is completed, the method further includes:

[0013] The data files whose formats have been converted are stored in a preset fixed file directory.

[0014] In a specific implementation of the present invention, storing the data file after format conversion in a preset fixed file directory includes:

[0015] The format conversion of the data files is stored in a fixed file directory set according to the preset downgrade storage strategy. In a specific embodiment of the present invention, the format conversion of the stored data files and the data files to be stored is performed in combination with the second erasure code formats corresponding to the pre-configured storage levels and the preset storage configuration priorities at each level, including:

[0016] In combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priority in which the storage granularity size is inversely proportional to the priority level, the formats of the stored data files and the data files to be stored are converted.

[0017] A device for storing upper-layer application data of distributed storage as erasure codes, comprising:

[0018] The configuration name reading module is used to read the preset data file storage configuration name when the service is initialized;

[0019] A judgment module, used to judge whether the data file storage configuration name is a configuration name within the merge strategy;

[0020] A first format conversion module is used for, when it is determined that the data file storage configuration name is a configuration name within the merge strategy, writing the target data file of the preset time range into a pre-created temporary subdirectory according to the time layer, and performing format conversion on the target data file according to the preset first erasure code format; wherein the temporary subdirectory is a subdirectory created under the temporary directory of the column family;

[0021] The second format conversion module is used to convert the formats of stored data files and data files to be stored in combination with the second erasure code formats corresponding to the pre-configured storage at each level and the preset storage configuration priorities at each level when it is determined that the data file storage configuration name is not a configuration name within the merge strategy.

[0022] In a specific embodiment of the present invention, it also includes:

[0023] The file storage module is used to store the data files after the format conversion is completed in a preset fixed file directory.

[0024] In a specific implementation of the present invention, the file storage module is specifically a module that stores the data files after format conversion in a fixed file directory set according to a preset degradation storage policy.

[0025] In a specific embodiment of the present invention, the second format conversion module is specifically a module for format conversion of stored data files and to-be-stored data files in combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priority in which the preset storage granularity size is inversely proportional to the priority level.

[0026] A device for storing upper-layer application data of distributed storage as erasure codes, comprising:

[0027] Memory for storing computer programs;

[0028] A processor is used to implement the steps of the method of storing upper-layer application data of distributed storage as erasure codes as described above when executing the computer program.

[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for storing upper-layer application data of distributed storage as erasure codes as described above.

[0030] The method for storing upper-layer application data of distributed storage as erasure codes provided by the present invention, when the service is initialized, reads a preset data file storage configuration name; determines whether the data file storage configuration name is a configuration name within a merge strategy; if so, writes a target data file within a preset time range into a pre-created temporary subdirectory according to time stratification, and performs format conversion on the target data file according to a preset first erasure code format; wherein; the temporary subdirectory is a subdirectory created under a temporary directory of a column family; if not, performs format conversion on stored data files and data files to be stored in combination with the pre-configured second erasure code formats corresponding to each level of storage and the preset storage configuration priorities at each level.

[0031] It can be seen from the above technical solution that by presetting the data file storage configuration name, it is distinguished according to the data file storage configuration name whether to perform erasure code conversion and storage on data files within a preset time range, or to perform erasure code conversion and storage on all stored data files and data files to be stored. And the erasure code format for erasure code conversion and storage is pre-set, and the data file is automatically formatted according to the corresponding erasure code conversion format. Compared with the method of manually performing erasure code format conversion, the present invention flexibly and conveniently specifies part or all of the data as erasure code format by changing the hbase configuration, thereby reducing storage costs and management complexity.

[0032] Correspondingly, the present invention also provides an apparatus, device and computer-readable storage medium for storing upper-layer application data of distributed storage as erasure codes corresponding to the above-mentioned method for storing upper-layer application data of distributed storage as erasure codes, which has the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A flowchart of an implementation of a method for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention;

[0035] Figure 2 Another implementation flow chart of the method for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention;

[0036] Figure 3 It is a structural block diagram of a device for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention;

[0037] Figure 4 It is a structural block diagram of a device for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention;

[0038] Figure 5 A schematic diagram of the specific structure of a device for storing upper-layer application data of a distributed storage as an erasure code is provided in this embodiment. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 , Figure 1 This is a flowchart of an implementation of a method for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention. The method may include the following steps:

[0041] S101: When the service is initialized, a preset data file storage configuration name is read.

[0042] Pre-set the data file storage configuration name. You can set the library, table, and column family level configuration names (general configuration such as hbase.hstore.block.ec.policy), and you can also set the configuration name within the merge (compaction) strategy (that is, specific configuration). And configure the client command (hbase shell statement, such as EC_POLICY). When the service is initialized, read the preset data file storage configuration name.

[0043] S102: Determine whether the data file storage configuration name is a configuration name within the merge strategy, if so, execute step S103, if not, execute step S104.

[0044] After reading the data file storage configuration name, determine whether the data file storage configuration name is a configuration name within the merge strategy. If so, it means that the current configuration is to perform erasure code conversion and storage on data files within a preset time range, and execute step S103. If not, it means that the current configuration is to perform erasure code conversion and storage on all stored data files and data files to be stored, and execute step S104.

[0045] S103: writing the target data files within the preset time range into a pre-created temporary subdirectory according to the time layer, and converting the format of the target data files according to a preset first erasure code format.

[0046] The temporary subdirectory is a subdirectory created under the temporary directory of the column family.

[0047] When erasure coding is converted and stored for data files within a preset time range according to time stratification, a marked temporary subdirectory is created in advance under the temporary directory of the column family, and the first erasure code format for erasure coding of data files within the preset time range is pre-set. When the data file storage configuration name is determined to be a configuration name within the merge strategy, it means that the current configuration is to perform erasure coding conversion and storage for data files within the preset time range. When the data file (i.e., hfile file) that reaches the corresponding time layer or the specified time range triggers a merge (compact) operation, the target data file within the preset time range is written to the pre-created temporary subdirectory according to the time stratification, and the target data file is format-converted according to the preset first erasure code format.

[0048] S104: Perform format conversion on stored data files and to-be-stored data files in combination with the pre-configured second erasure code formats corresponding to each level of storage and the preset configuration priorities of each level of storage.

[0049] Pre-configure the conversion priority for format conversion of all stored data files and data files to be stored in each level of storage, and set the second erasure code formats corresponding to each level of storage. When it is determined that the data file storage configuration name is not a configuration name within the merge strategy, it means that the current configuration is to perform erasure code conversion storage on all stored data files and data files to be stored, and perform format conversion on the stored data files and data files to be stored in combination with the pre-configured second erasure code formats corresponding to each level of storage and the preset storage configuration priorities at each level.

[0050] It should be noted that there is no difference in size or order between the first erasure code format and the first and second erasure code formats in each second erasure code format, and it is only for illustrating that the expansion rate and fault tolerance rate allowed by the configured data files are different.

[0051] The present invention flexibly and conveniently specifies part or all of the data as erasure code format by changing the configuration of the hbase database. This makes the hbase database more friendly to the erasure code format. According to the different needs of storage and query activity, the whole database level, table level, column family level, or different life cycles of data in the same column family are set as the storage format of the erasure code strategy, so that the data of low-frequency queries occupies less storage space, and the storage cost and management complexity are reduced without reducing the overall performance.

[0052] It can be seen from the above technical solution that by presetting the data file storage configuration name, it is distinguished according to the data file storage configuration name whether to perform erasure code conversion and storage on data files within a preset time range, or to perform erasure code conversion and storage on all stored data files and data files to be stored. And the erasure code format for erasure code conversion and storage is pre-set, and the data file is automatically formatted according to the corresponding erasure code conversion format. Compared with the method of manually performing erasure code format conversion, the present invention flexibly and conveniently specifies part or all of the data as erasure code format by changing the hbase configuration, thereby reducing storage costs and management complexity.

[0053] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0054] See also Figure 2 , Figure 2 This is another implementation flow chart of a method for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention. The method may include the following steps:

[0055] S201: When the service is initialized, read the preset data file storage configuration name.

[0056] S202: Determine whether the data file storage configuration name is a configuration name within the merge strategy, if so, execute step S203, if not, execute step S204.

[0057] S203: writing the target data files within the preset time range into a pre-created temporary subdirectory according to the time layer, and converting the format of the target data files according to a preset first erasure code format.

[0058] The temporary subdirectory is a subdirectory created under the temporary directory of the column family.

[0059] S204: Perform format conversion on stored data files and to-be-stored data files in combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priorities in which the storage granularity is inversely proportional to the priority.

[0060] The configuration priority is pre-set so that the storage granularity is inversely proportional to the priority level, such as setting the priority level to configuration within the merge strategy (i.e., time-tiered configuration) > column family level configuration > table level configuration > full database level configuration. When it is determined that the data file storage configuration name is not a configuration name within the merge strategy, the format of the stored data files and the data files to be stored are converted in combination with the second erasure code format corresponding to each level of pre-configured storage and the preset configuration priority of the storage granularity being inversely proportional to the priority level. By setting the configuration priority level so that the storage granularity is inversely proportional to the priority level, the accuracy of the conversion of the erasure code format of each level of storage is improved.

[0061] S205: After the format conversion is completed, the data file whose format conversion is completed is stored in a preset fixed file directory.

[0062] A fixed file directory to be exported after the data file format conversion is completed is pre-set in the temporary directory. After the data file format conversion is completed, the data file with the format conversion completed is stored in the preset fixed file directory, thereby completing the format conversion and merged storage of the data file.

[0063] In a specific implementation of the present invention, step S205 may include the following steps:

[0064] The data files that have completed the format conversion are stored in a fixed file directory set according to the preset downgrade storage policy. When the service is initialized, it is detected whether the erasure code strategy and the heterogeneous storage strategy coexist. If so, the heterogeneous storage space corresponding to the fixed file directory is converted to a storage space with a unified structure according to the preset downgrade storage policy. When it is necessary to store the files after the format conversion is completed, the data files that have completed the format conversion are stored in a fixed file directory set according to the preset downgrade storage policy. When the erasure code strategy and the heterogeneous storage strategy coexist, downgrade storage is performed to resolve the configuration conflict between the erasure format conversion storage and the heterogeneous storage strategy.

[0065] Corresponding to the above method embodiment, the present invention also provides a device for storing upper-layer application data of distributed storage as erasure codes. The device for storing upper-layer application data of distributed storage as erasure codes described below and the method for storing upper-layer application data of distributed storage as erasure codes described above can be referenced to each other.

[0066] See also Figure 3 , Figure 3 This is a structural block diagram of a device for storing upper-layer application data of distributed storage as erasure codes in an embodiment of the present invention. The device may include:

[0067] The configuration name reading module 31 is used to read the preset data file storage configuration name when the service is initialized;

[0068] A judgment module 32 is used to judge whether the data file storage configuration name is a configuration name within the merge strategy;

[0069] The first format conversion module 33 is used to write the target data file in the preset time range into a pre-created temporary subdirectory according to the time layer when it is determined that the data file storage configuration name is the configuration name within the merge strategy, and perform format conversion on the target data file according to the preset first erasure code format; wherein the temporary subdirectory is a subdirectory created under the temporary directory of the column family;

[0070] The second format conversion module 34 is used to convert the formats of the stored data files and the data files to be stored in combination with the second erasure code formats corresponding to the pre-configured storage levels and the preset storage configuration priorities of each level when it is determined that the data file storage configuration name is not a configuration name within the merge strategy.

[0071] It can be seen from the above technical solution that by presetting the data file storage configuration name, it is distinguished according to the data file storage configuration name whether to perform erasure code conversion and storage on data files within a preset time range, or to perform erasure code conversion and storage on all stored data files and data files to be stored. And the erasure code format for erasure code conversion and storage is pre-set, and the data file is automatically formatted according to the corresponding erasure code conversion format. Compared with the method of manually performing erasure code format conversion, the present invention flexibly and conveniently specifies part or all of the data as erasure code format by changing the hbase configuration, thereby reducing storage costs and management complexity.

[0072] In a specific embodiment of the present invention, the device may further include:

[0073] The file storage module is used to store the data files after the format conversion is completed in a preset fixed file directory.

[0074] In a specific implementation of the present invention, the file storage module is specifically a module that stores the data files after format conversion in a fixed file directory set according to a preset degradation storage policy.

[0075] In a specific embodiment of the present invention, the second format conversion module 34 is specifically a module for converting the formats of stored data files and data files to be stored in combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priority in which the preset storage granularity size is inversely proportional to the priority level.

[0076] Corresponding to the above method embodiment, see Figure 4 , Figure 4This is a schematic diagram of a device for storing upper-layer application data of distributed storage provided by the present invention as erasure codes, and the device may include:

[0077] A memory 332, for storing computer programs;

[0078] The processor 322 is configured to implement the steps of the method for storing upper-layer application data in distributed storage as erasure codes in the above method embodiment when executing a computer program.

[0079] For details, please refer to Figure 5 , Figure 5 A schematic diagram of the specific structure of a device for storing upper-layer application data in distributed storage as an erasure code is provided for this embodiment. The device for storing upper-layer application data in distributed storage as an erasure code may have relatively large differences due to different configurations or performances, and may include a processor (central processing unit, CPU) 322 (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332, and execute a series of instruction operations in the memory 332 on the device 301 for storing upper-layer application data in distributed storage as an erasure code.

[0080] The device 301 storing upper-layer application data of distributed storage as erasure codes may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0081] The steps in the method for storing upper-layer application data in distributed storage as erasure codes described above may be implemented by the structure of a device for storing upper-layer application data in distributed storage as erasure codes.

[0082] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0083] When the service is initialized, read the preset data file storage configuration name; determine whether the data file storage configuration name is a configuration name within the merge strategy; if so, write the target data file in the preset time range into a pre-created temporary subdirectory according to the time layer, and convert the target data file format according to the preset first erasure code format; wherein; the temporary subdirectory is a subdirectory created under the temporary directory of the column family; if not, convert the format of the stored data files and the data files to be stored in combination with the second erasure code formats corresponding to the pre-configured storage levels and the preset storage configuration priorities at each level.

[0084] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0085] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0086] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0087] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, the present invention can also be improved and modified without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A method for storing upper-layer application data of distributed storage as erasure codes, characterized in that: include: When the service is initialized, read the preset data file storage configuration name; Determine whether the data file storage configuration name is a configuration name within the merge strategy; If so, write the target data file of the preset time range into a pre-created temporary subdirectory according to the time layer, and convert the format of the target data file according to the preset first erasure code format; wherein the temporary subdirectory is a subdirectory created under the temporary directory of the column family; If not, the formats of the stored data files and the data files to be stored are converted in combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priorities in which the storage granularity size is inversely proportional to the priority level.

2. The method for storing upper-layer application data of distributed storage as erasure codes according to claim 1, characterized in that: After the format conversion is completed, it also includes: The data files whose formats have been converted are stored in a preset fixed file directory.

3. The method for storing upper-layer application data of distributed storage as erasure codes according to claim 2, characterized in that: The converted data files are stored in the preset fixed file directory, including: The data files whose formats have been converted are stored in a fixed file directory set according to the preset downgrade storage strategy.

4. A device for storing upper-layer application data of distributed storage as erasure codes, characterized in that: include: The configuration name reading module is used to read the preset data file storage configuration name when the service is initialized; A judgment module, used to judge whether the data file storage configuration name is a configuration name within the merge strategy; A first format conversion module is used for, when it is determined that the data file storage configuration name is a configuration name within the merge strategy, writing the target data file of the preset time range into a pre-created temporary subdirectory according to the time layer, and performing format conversion on the target data file according to the preset first erasure code format; wherein the temporary subdirectory is a subdirectory created under the temporary directory of the column family; The second format conversion module is used to convert the formats of stored data files and data files to be stored in combination with the second erasure code formats corresponding to the pre-configured storage levels and the configuration priorities in which the storage granularity size is inversely proportional to the priority level when it is determined that the data file storage configuration name is not a configuration name within the merge strategy.

5. The device for storing upper-layer application data of distributed storage as erasure codes according to claim 4, characterized in that: Also includes: The file storage module is used to store the data files after the format conversion is completed in a preset fixed file directory.

6. The device for storing upper-layer application data of distributed storage as erasure codes according to claim 5, characterized in that: The file storage module is specifically a module that stores the data files after format conversion in a fixed file directory set according to a preset degradation storage strategy.

7. A device for storing upper-layer application data in distributed storage as erasure codes, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method for storing upper-layer application data of distributed storage as erasure codes as claimed in any one of claims 1 to 3 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method for storing upper-layer application data of distributed storage as erasure codes as claimed in any one of claims 1 to 3.

Images