Data downgrade storage method, electronic device and storage medium

Abstract

The present application discloses a data degradation storage method, an electronic device, and a storage medium. The data degradation storage method includes: obtaining the expiration time of the initial data file in the storage space, the initial data file is a stored data file that exceeds the default expiration period; in response to the expiration time of the initial data file being within the target preset time period, determining the erasure code rule corresponding to the target preset time period as the target erasure code rule of the initial data file, the target preset time period being one of several preset time periods, and different preset time periods corresponding to different erasure code rules; converting the format of the initial data file according to the target erasure code rule to obtain a target data file, the data volume of the target data file being less than the data volume of the initial data file. The above scheme can improve the utilization rate of the storage space.

Description

Technical Field

[0001] The present application relates to the field of storage technology, and in particular to a data degradation storage method, an electronic device and a storage medium. Background Art

[0002] When storing video recordings and pictures of important business scenarios, a multi-copy storage mode is usually used, which has a long storage period and high data reliability requirements. As time goes by, the data reliability of the stored data stored at the historical moment decreases, but the stored data will continue to occupy storage space, resulting in very low storage space utilization, thereby reducing data storage efficiency.

[0003] In view of the existing technical defects, how to provide a solution to improve the utilization rate of storage space is a technical problem that needs to be solved urgently by those skilled in the art. Summary of the invention

[0004] The present application at least provides a data degradation storage method, an electronic device and a storage medium.

[0005] The present application provides a data degradation storage method, comprising: obtaining an expiration time of an initial data file in a storage space, the initial data file being a stored data file that exceeds a default expiration period; in response to the expiration time of the initial data file being within a target preset time period, determining an erasure code rule corresponding to the target preset time period as a target erasure code rule for the initial data file, the target preset time period being one of a plurality of preset time periods, and different preset time periods corresponding to different erasure code rules; performing format conversion on the initial data file according to the target erasure code rule to obtain a target data file, wherein the data volume of the target data file is smaller than the data volume of the initial data file. The present application provides a data degradation storage device, including: an acquisition module, a determination module and a conversion module; the acquisition module is used to acquire the expiration time of an initial data file in a storage space, the initial data file being a stored data file that exceeds a default expiration period; the determination module is used to determine, in response to the expiration time of the initial data file being within a target preset time period, an erasure code rule corresponding to the target preset time period as a target erasure code rule for the initial data file, the target preset time period being one of several preset time periods, and different preset time periods corresponding to different erasure code rules; the conversion module is used to perform format conversion on the initial data file according to the target erasure code rule to obtain a target data file, wherein the data volume of the target data file is smaller than the data volume of the initial data file.

[0006] The present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned data degradation storage method.

[0007] The present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, the above-mentioned data degradation storage method is implemented.

[0008] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0011] Figure 1 This is a flow chart of an embodiment of the data downgrade storage method of the present application. Figure 1 ;

[0012] Figure 2 This is a flow chart of an embodiment of the data downgrade storage method of the present application. Figure 2 ;

[0013] Figure 3 It is a schematic diagram of the process of writing multiple copies of data in an embodiment of the data downgrade storage method of the present application;

[0014] Figure 4 It is a schematic diagram of a process of converting a multi-copy data type into an erasure code data type in an embodiment of a data degradation storage method of the present application;

[0015] Figure 5 It is a schematic diagram of a process of converting an erasure code data type into an erasure code data type in an embodiment of the data degradation storage method of the present application;

[0016] Figure 6 It is a structural diagram of an embodiment of a data downgrade storage device of the present application;

[0017] Figure 7 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0018] Figure 8 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0019] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0020] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0021] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0022] The present application provides some data downgrade storage methods and data downgrade storage devices. The application scenarios of the data downgrade storage method include but are not limited to video recording and picture storage processes in important business scenarios. The executor of the data downgrade storage method may be a data downgrade storage device. For example, the data downgrade storage device may be arranged in a terminal device or a server or other processing device, wherein the terminal device may be a device for data downgrade storage, a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, etc. In some possible implementations, the data downgrade storage method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0023] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of the data downgrade storage method of the present application. Figure 1 Specifically, the data downgrade storage method may include the following steps:

[0024] Step S11: Obtain the expiration time of the initial data file in the storage space.

[0025] The initial data file is a stored data file that exceeds the default expiration period.

[0026] The storage space is a storage device capable of storing target data. The type of the storage space may be local storage, hard disk drive, solid state drive, network attached storage, cloud storage, etc. Different initial data files may correspond to different default expiration periods. In this application, the default expiration period of the data files stored in the storage space is the same. The default expiration period is the period after the default time after the data file is stored in the storage space. According to the requirements of the degraded storage accuracy, the value of the default time can be dynamically set, which is not limited here. The expiration time of the initial data file is used to represent the difference between the current time and the default expiration period of the initial data file. The difference can be measured in days or hours, etc. Information as a unit of time. It can be understood that the storage space includes several metadata files. The stored data file is any metadata file in the storage space. The initial data file in the above step S11 can be any metadata file of several metadata that exceeds the default expiration period. This application takes the data degraded storage of any initial data file in the storage space as an example. In addition, when all the initial data files in the storage space are stored in data downgrade, they can be processed in parallel to improve the efficiency of the data downgrade storage of each initial data file, and the method of storing all the initial data files in the storage space in data downgrade is not repeated here. For example, the expiration time of the initial data file A, the initial data file B, and the initial data file C are respectively 2 days, 16 days, and 35 days beyond the default expiration period.

[0027] The way to obtain the expiration time of the initial data file in the storage space may be to determine the expiration time of the initial data file in the storage space in response to the expiration time acquisition instruction. The way may also be to directly use the provided API to obtain the initial data file and the expiration time of the initial data file.

[0028] Step S12: in response to the expiration time of the initial data file being within the target preset time period, determining the erasure coding rule corresponding to the target preset time period as the target erasure coding rule of the initial data file.

[0029] The target preset time period is one of several preset time periods, and different preset time periods correspond to different erasure code rules.

[0030] The time after the default expiration period is pre-divided into a plurality of preset time periods. Each preset time period corresponds to an erasure code rule. The erasure code rule includes at least one of the corresponding preset time period, the data type of the initial data file, the data type of the target data file, the number of storage nodes required for the target data file, and a set storage pool.

[0031] The data type of the initial data file includes a multi-copy data type or an erasure code data type. The target data file is a data file obtained after format conversion of the initial data file based on the target erasure code rule. The data type of the target data file includes an erasure code data type. The number of storage nodes required for the target data file may be at least two. The farther the preset time period corresponding to the erasure code rule is from the default expiration period, the more storage nodes are required for the target data file. The storage pool is set to include a hot data storage pool or a cold data storage pool. The corresponding preset time period included in the erasure code rule is used to indicate that the expiration time of the initial data file belonging to the erasure code rule is within the preset time period of the erasure code rule. The data type of the initial data file included in the erasure code rule is used to indicate the data type of the initial data file belonging to the erasure code rule. The data type of the target data file included in the erasure code rule is used to indicate the data type of the data file obtained after the format conversion of the initial data file belonging to the erasure code rule. The number of storage nodes required for the target data file included in the erasure code rule is used to indicate the number of storage nodes required when the format conversion of the initial data file belonging to the erasure code rule is performed. The set storage pool included in the erasure code rule is used to indicate the storage pool into which the data file obtained after the format conversion of the initial data file belonging to the erasure code rule should be written.

[0032] Exemplarily, the content of the first type of erasure code rule includes that the preset time period corresponding to the erasure code rule is the first type of preset time period, the data type of the initial data file is the multi-copy data type, the data type of the target data file is the erasure code data type, the number of storage nodes required for the target data file is M1, and the storage pool is set to be at least one of the hot data storage pool. Exemplarily, the content of the second type of erasure code rule includes that the preset time period corresponding to the erasure code rule is the second type of preset time period, the data type of the initial data file is the erasure code data type, the data type of the target data file is the erasure code data type, the number of storage nodes required for the target data file is M2, and the storage pool is set to be at least one of the cold data storage pool. Among them, the first type of preset time period is before the second type of preset time period. Compared with the second type of preset time period, the first type of preset time period is closest to the default expiration period. The number of storage nodes required for the target data file in the first type of erasure code rule is M1, which is less than the number of storage nodes required for the target data file in the second type of erasure code rule is M2. Among them, the first type of erasure code rule includes a plurality of first erasure code rules, and the first type of preset time period includes at least one first preset time period. The number of first erasure codes in several first erasure code rules is the same as the number of first preset time periods. The farther the first preset time period corresponding to the erasure code rule is from the default expiration period, the more storage nodes are required for the target data file. Among them, the first erasure code rules are the same except that the first preset time settings are different and the number of storage nodes required for the target data file is different. The second type of erasure code rules includes several second erasure code rules. The settings of each second erasure code rule can refer to the settings of each first erasure code rule, which will not be repeated here. It can be understood that according to the requirements of degraded storage accuracy, the number and time scale of the preset time periods can be dynamically set and divided, which is not limited here.

[0033] In some application scenarios, the above step S12 may be to determine the relationship between the expiration time of the initial data file and the first type of preset time period and the second type of preset time period. The target preset time period may be the first type of preset time period or the second type of preset time period.

[0034] In response to the expiration time of the initial data file being within the first preset time period, a first erasure coding rule corresponding to the first preset time period is determined as a target erasure coding rule for the initial data file. In response to the expiration time of the initial data file being within the second preset time period, a second erasure coding rule corresponding to the second preset time period is determined as a target erasure coding rule for the initial data file.

[0035] Step S13: convert the format of the initial data file according to the target erasure coding rule to obtain the target data file.

[0036] The data size of the target data file is smaller than the data size of the initial data file.

[0037] The format conversion may be to convert the initial data file into the target data file based on the number of storage nodes required in the target erasure code rule and the data type of the target data file.

[0038] In some application scenarios, the above step S12 may be in response to the expiration time of the initial data file being within the first preset time period, determining the first type of erasure code rule corresponding to the first preset time period as the target erasure code rule of the initial data file. The above step S13 may be format conversion of the initial data file according to the first type of erasure code rule to obtain the target data file. In other application scenarios, the above step S12 may be in response to the expiration time of the initial data file being within the second preset time period, determining the second type of erasure code rule corresponding to the second preset time period as the target erasure code rule of the initial data file. The above step S13 may be format conversion of the initial data file according to the second type of erasure code rule to obtain the target data file.

[0039] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0040] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of the data downgrade storage method of the present application. Figure 2 .

[0041] In some embodiments, the storage space includes a hot data storage pool and a cold data storage pool, and the above step S13 may include the following steps: Step S21: Based on the data type of the initial data file, determine the storage pool to which the target data file should be written. The storage pool is a hot data storage pool or a cold data storage pool. Step S22: Convert the initial data file into target erasure coded data using the target erasure code rule. Step S23: Write the target erasure coded data into the storage pool to obtain the target data file.

[0042] The storage space includes a hot data storage pool and a cold data storage pool. The hot data storage pool can store data files of the multi-copy data type and data files of the erasure code data type. The cold data storage pool can store data files of the erasure code data type. The importance of the data files in the hot data storage pool is higher than the importance of the data files in the cold data storage pool. The expiration time of the data files in the hot data storage pool is lower than the expiration time of the data files in the cold data storage pool.

[0043] The data type of the initial data file may be a multi-copy data type or an erasure code data type. In the case where the data type of the initial data file is different, the storage pool to which the target data file should be written is different. The above step S21 may be to first determine the data type of the initial data file. Then, based on the determined data type of the initial data file, determine the storage pool to which the target data file should be written. Specifically, the above step S21 may be that in the case where the data type of the initial data file is a multi-copy data type, the storage pool to which the target data file should be written is a hot data storage pool. The above step S21 may be that in the case where the data type of the initial data file is an erasure code data type, the storage pool to which the target data file should be written is a cold data storage pool.

[0044] In some other application scenarios, the above step S21 may also be to determine the storage pool to which the target data file should be written based on the target erasure code rule and / or the data type of the initial data file. Specifically, in the case where the above step S21 is to determine the storage pool to which the target data file should be written based on the target erasure code rule, the target erasure code rule includes at least one of the data type of the initial data file and the set storage pool. In the case where the above step S21 is to determine the storage pool to which the target data file should be written based on the target erasure code rule and the data type of the initial data file, the target erasure code rule may include the data type of the initial data file and / or the set storage pool.

[0045] The target erasure code data is the erasure code data obtained by converting the initial data file according to the target erasure code rule. The process of converting the initial data file may be the process of encoding the initial data file according to the target erasure code rule to obtain the erasure code data. The target erasure code data includes the data blocks after data segmentation and the generated check blocks. The erasure code rule may be an (n, k) erasure code, which specifically divides the data into k data blocks and generates nk check blocks, for a total of n blocks. Among the n blocks, as long as k blocks are available, the original data can be restored. The above step S22 may be a process of determining the need for the target erasure code data according to the target erasure code rule.

[0046] In some application scenarios, when the data type of the initial data file is multiple copies of data, the initial data file, that is, the same data, stores multiple copies of data on multiple storage nodes to improve data reliability and access speed. If the initial data file is multiple copies of data, the process of converting to erasure code data may be: the above step S22 may be to use the target erasure code rule to perform data segmentation on any copy of the data corresponding to the initial data file and generate a check block to obtain the segmented data blocks and check blocks. The target erasure code rule may indicate the number of segmented data blocks. The above step S23 may be to store the segmented data blocks and check blocks respectively in the storage pool to which the target data file should be written. Specifically, the target erasure code rule is used to segment the copy data of the initial data file on any storage node, and the segmented data blocks obtained are k data blocks. The target erasure code rule is used to generate nk check blocks for the initial data file. The k data blocks and nk check blocks are stored on different storage nodes. The location of the different storage nodes in the storage space and the number of storage nodes required are determined by the target erasure code rule. The locations of different storage nodes in the storage space are storage pools to which target data files should be written.

[0047] In other application scenarios, when the data type of the initial data file is erasure coded data, the initial data file may also be data that has been encoded with erasure coding, and the conversion process of the initial data file may involve re-encoding or adjusting the existing erasure coding strategy. If the initial data file is already erasure coded data, but needs to be converted to a new erasure coding rule, the process is as follows: the above step S22 may be: using the initial erasure coding rule to decode the initial data file and restore the original data. Among them, the initial erasure coding rule may be the erasure coding rule used in the process of obtaining the initial data file. The number of storage nodes required for the initial erasure coding rule is less than the number of storage nodes required for the target erasure coding rule. Specifically, the initial erasure coding rule is used to decode several data blocks and several check blocks of the initial data file to obtain the restored original data. The target erasure coding rule is used to re-encode the original data to generate new data blocks and new check blocks. The above step S23 may be: storing the new data blocks and new check blocks in the storage pool to which the target data file should be written, respectively. Specifically, the new data blocks and new check blocks are stored on different storage nodes. The locations of the different storage nodes in the storage space and the number of storage nodes required are determined by the target erasure code rule. The locations of the different storage nodes in the storage space are the storage pools to which the target data files should be written.

[0048] It can be considered that the conversion of the initial data file format into the target data file can be completed by using the target erasure code rule to convert the initial data file into the target erasure code data, and writing the target erasure code data into the storage pool where the determined target data file should be written. This can ensure the integrity of the initial data file being downgraded to the target data file. The data volume of the target data file is smaller than that of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby freeing up the memory of the storage space.

[0049] In some embodiments, the storage pool includes several candidate storage nodes. Before the above step S23, the above data degradation storage method may further include the following steps: selecting a target number of storage nodes from the several candidate storage nodes as target storage nodes. The target number is obtained by the target erasure code rule. The location of the storage pool where the target storage node is located is used as the target storage location of the initial data file.

[0050] The number of storage nodes required for the target data file included in the target erasure code rule is determined as the target number. The candidate storage nodes included in the storage pool are storage nodes that can write data. The target storage node is the storage node that can write data to the target erasure code data in the storage pool. The location of the storage pool where the target storage node is located is the location of the target storage node in the storage pool. The target storage location of the initial data file is used to indicate the location of the storage node in the storage pool where the target erasure code data is written.

[0051] The above-mentioned selection method of selecting a target number of storage nodes from a number of candidate storage nodes can be random selection, polling selection, selection of the candidate storage node with the smallest load based on node load, and selection of the candidate storage node with the best node performance based on node performance.

[0052] In some application scenarios, when the target erasure coding rule is the first type of erasure coding rule, several candidate storage nodes are storage nodes in the hot data storage pool. The target number is determined based on the first type of erasure coding rule. The target number of storage nodes is selected from the candidate storage nodes in the hot data storage pool according to the above selection method as the target storage nodes. The target storage node is a storage node in the hot data storage pool. The location of the target storage node in the hot data storage pool is used as the target storage location of the initial data file.

[0053] In some other application scenarios, when the target erasure code rule is the second type of erasure code rule, the plurality of candidate storage nodes are storage nodes in the cold data storage pool. The target number is determined based on the second type of erasure code rule. The target number of storage nodes is selected from the plurality of candidate storage nodes in the cold data storage pool as the target storage nodes according to the above selection method. The target storage nodes are storage nodes in the cold data storage pool. The location of the target storage nodes in the cold data storage pool is used as the target storage location of the initial data file.

[0054] In some embodiments, the above step S23 may include the following steps: writing the target erasure code data into the target storage location to obtain a target data file.

[0055] The target storage location includes the location of each storage node on the target storage node in the storage pool. A plurality of data blocks and a plurality of check blocks in the target erasure coded data are stored in each storage node on the target storage node.

[0056] It can be considered that the target erasure coded data is written to the target storage location in the storage pool where the initial data file should be written. The storage node corresponding to the target storage location can store data files of lower importance, which can ensure that the initial data file is downgraded to the target data file and release the memory of the storage space.

[0057] In some application scenarios, after the above step S21, the storage pool is divided to obtain the cache storage area of ​​the storage pool corresponding to the initial data file. The above step S23 may include: writing the target erasure code data into the cache storage area of ​​the storage pool to obtain the target data file, so that when the target data file is subsequently downgraded for storage, it is not necessary to read the target data file from the entire storage pool, but only read the target data file from the cache storage area of ​​the storage pool, thereby improving the speed of reading the data file. The storage pool can be divided into multiple cache storage areas according to the number of preset time periods corresponding to various erasure code rules, so that the storage time of any data file exceeds the default expiration period and can be sequentially downgraded and stored in each cache storage area of ​​the hot data storage pool and each cache storage area of ​​the cold data storage pool according to the period related to each preset time period, so that the memory in the storage space occupied by the data file can be continuously reduced according to the period related to each preset time period, and each downgraded storage only needs to read the data file from the previous cache storage area to facilitate the management of data downgraded storage.

[0058] In some embodiments, the above data degraded storage method may further include the following steps: in response to the storage pool being a hot data storage pool, the hot data storage pool is divided into a cache storage area.

[0059] The cache storage area is used to represent a storage area obtained through area division in the storage pool to which the determined initial data file should be written.

[0060] In the case where the data type of the initial data file is a multiple-copy data type, the storage pool to which the target data file should be written is a hot data storage pool.

[0061] In some embodiments, the above step S23 may include the following steps: writing the target erasure coded data into a cache storage area in a hot data storage pool to obtain a target data file.

[0062] Write a number of data blocks and a number of check blocks in the target erasure coded data into a cache storage area in a hot data storage pool to obtain a target data file.

[0063] In some other application scenarios, in response to the storage pool being a cold data storage pool, the cold data storage pool is divided to obtain a cache storage area. The target erasure coded data is written into the cache storage area in the cold data storage pool to obtain a target data file. Specifically, a plurality of data blocks and a plurality of check blocks in the target erasure coded data are written into the cache storage area in the cold data storage pool to obtain a target data file.

[0064] It is understandable that, when the storage pool is a hot data storage pool, the location of the target storage node in the hot data storage pool, that is, the target storage location of the initial data file is the cache storage area in the hot data storage pool. When the storage pool is a cold data storage pool, the location of the target storage node in the cold data storage pool, that is, the target storage location of the initial data file is the cache storage area in the cold data storage pool.

[0065] In some embodiments, the data type of the initial data file includes a multi-copy data type or an erasure code data type, the plurality of preset time periods include a first type of preset time period related to the multi-copy data type and a second type of preset time period related to the erasure code data type, the first type of preset time period is before the second type of preset time period, and the above step S12 may include the following steps:

[0066] In response to the target preset time period being the first type of preset time period, the erasure coding rule corresponding to the multi-copy data type is determined as the target erasure coding rule. Or, in response to the target preset time period being the second type of preset time period, the erasure coding rule corresponding to the erasure coding data type is determined as the target erasure coding rule.

[0067] The target preset time is the preset time period in which the expiration time of the initial data file is located. The erasure code rule corresponding to the multi-copy data type is the first type of erasure code rule mentioned above. The erasure code rule corresponding to the erasure code data type is the second type of erasure code rule mentioned above. The erasure code rule corresponding to each preset time period in the plurality of preset time periods is related to the data type of the initial data file. When the initial data file is of the multi-copy data type, the preset time period in which the expiration time of the initial data file is located is the preset time period corresponding to the erasure code rule related to the data type of the initial data file.

[0068] In some application scenarios, when it is determined that the preset time period in which the expiration time of the initial data file is located is the first type of preset time period related to the multi-copy data type, the erasure code rule corresponding to the multi-copy data type is determined as the target erasure code rule, that is, the first type of erasure code rule corresponding to the multi-copy data type is determined as the target erasure code rule. It can be understood that the first type of preset time period corresponds to the data type of the initial data file being multi-copy data. Exemplarily, in the case where the first type of preset time period includes only one preset time period, the first type of erasure code rule includes one erasure code rule, and the erasure code rule corresponding to the multi-copy data type is directly determined as the target erasure code rule.

[0069] In some other application scenarios, when it is determined that the preset time period in which the expiration time of the initial data file is located is the second type of preset time period related to the erasure code data type, the erasure code rule corresponding to the erasure code data type is determined as the target erasure code rule, that is, the second type of erasure code rule corresponding to the erasure code data type is determined as the target erasure code rule. It can be understood that the second type of preset time period corresponds to the data type of the initial data file being erasure code data. Exemplarily, in the case where the second type of preset time period includes at least one second preset time period, for example, the second type of preset time period includes 2 second preset time periods, the first second preset time period is before the non-first second preset time period, the second type of erasure code rules includes 2 second erasure code rules, the first second preset time period corresponds to the first second erasure code rule in the second type of erasure code rules, and the non-first second preset time period corresponds to the non-first second erasure code rule in the second type of erasure code rules, the above-mentioned determination of the second type of erasure code rule corresponding to the erasure code data type as the target erasure code rule includes: taking one of the at least one second preset time period in which the expiration time of the initial data file is located as the target second preset time period, and taking the erasure code rule corresponding to the target second preset time period as the target erasure code rule. Specifically, in response to one of the at least one second preset time period in which the expiration time of the initial data file is located as the first second preset time period, taking the first second erasure code rule corresponding to the first second preset time period as the target erasure code rule.

[0070] It can be considered that determining the erasure code rule corresponding to the target preset time period by the preset time period of the expiration time of the initial data file can improve the accuracy of the determined erasure code rule, thereby improving the efficiency of data degraded storage.

[0071] In some embodiments, after the above step S13, the above data degraded storage method may further include the following steps: determining whether the initial data file is successfully written. In response to the initial data file being successfully written, deleting the initial data file.

[0072] After the above step S13, the writing result related to the initial data file is fed back. The writing result includes: successful writing or writing failure. In the case where the writing result is successful writing, the successful writing of the initial data file can represent that the target erasure code data obtained by the conversion of the initial data file is successfully written into the storage pool, the cache storage area and / or the target storage location to obtain the target data file. Specifically, it is determined whether the writing result of the initial data file is successful writing. In response to the writing result of the initial data file being successful writing, the initial data file is deleted.

[0073] In the case where the write result is a write failure, the write failure of the initial data file can represent that the target erasure code data obtained after the conversion of the initial data file has not been successfully written into the storage pool, the cache storage area and / or the target storage location. In response to the write result of the initial data file being a write failure, the write result related to the initial data file is fed back at a preset frequency. In some application scenarios, the write result related to the initial data file is fed back at a preset frequency until the write result of the initial data file is successfully written, and the initial data file is deleted. In other application scenarios, the write result related to the initial data file is fed back at a preset frequency, and the write result of the initial data file exceeds the threshold time, and the write result of the initial data file is still a write failure, and the above step S12 is re-executed.

[0074] It can be considered that in response to successful writing of the initial data file, deleting the initial data file can ensure that the metadata of the same data file in the storage space is only the data file after downgraded storage, and can ensure the reliability of the metadata of each data file in the storage space.

[0075] In some embodiments, the above-mentioned data degradation storage method may further include the following steps: obtaining a rule configuration instruction. The rule configuration instruction is used to indicate the rule configuration information of each erasure code rule, and each rule configuration information includes at least one of a preset time period corresponding to each erasure code rule, a data type of an initial data file corresponding to each erasure code rule, and a number of storage nodes required for each erasure code rule. Based on the rule configuration instruction, each erasure code rule is generated.

[0076] The rule configuration instruction is an instruction generated in response to an operation command related to the configuration of the erasure code rule on the display interface. The rule configuration information of each erasure code rule is obtained based on the operation command related to the configuration of the erasure code rule on the display interface.

[0077] The corresponding preset time period included in the erasure code rule is used to indicate that the expiration time of the initial data file belonging to the erasure code rule is within the preset time period of the erasure code rule. The data type of the initial data file included in the erasure code rule is used to indicate the data type of the initial data file belonging to the erasure code rule. The data type of the target data file included in the erasure code rule is used to indicate the data type of the data file obtained after the format conversion of the initial data file belonging to the erasure code rule. The number of storage nodes required for the target data file included in the erasure code rule is used to indicate the number of storage nodes required when the initial data file belonging to the erasure code rule is format converted. Specifically, each rule configuration information also includes the number of data blocks and the number of check blocks corresponding to the target erasure code data in the case of the number of required storage nodes.

[0078] Each rule configuration information includes a storage pool set in each erasure code rule, and each set storage pool is used to indicate a storage pool where a data file obtained after format conversion of an initial data file belonging to the erasure code rule should be written.

[0079] Based on the rule configuration instruction, the generation of each erasure code rule can be to parse the rule configuration instruction to obtain each rule configuration information. Each rule configuration information is assembled with the erasure code rule template to obtain each erasure code rule. Each rule configuration information is assembled with the erasure code rule template to obtain each erasure code rule that can specify the number of data blocks, the number of check blocks, the selection strategy of candidate storage nodes, such as load balancing, etc., and the recovery threshold, that is, the data can still be recovered in the case of how many blocks are lost.

[0080] It can be considered that different types of erasure code rules can be flexibly generated through dynamically set rule configuration instructions. Each erasure code rule can adjust the number of data blocks, the number of check blocks, and the storage node selection strategy according to specific needs, thereby optimizing the performance and reliability of data degraded storage.

[0081] In some embodiments, the above-mentioned data degraded storage method may further include the following steps: first, obtain a flow control instruction for the target thread. The flow control instruction is used to set the number of files processed by the target thread each time. The target thread includes a data reading thread, a data conversion thread and / or a data cleaning thread. The data reading thread is used to obtain the initial data file, the data conversion thread is used to convert the format of the initial data file, and the data cleaning thread is used to delete the initial data file. Then, flow control processing is performed on the target thread based on the flow control instruction.

[0082] The data reading thread can read the initial data file. The data conversion thread can convert the format of the initial data file according to the target erasure code rule of the initial data file to obtain the target data file. The data cleaning thread can respond to the successful writing of the initial data file and delete the initial data file.

[0083] The flow control instruction is an instruction generated in response to an operation command on the display interface to select one of low-speed flow control, medium-speed flow control and high-speed flow control.

[0084] Flow control is required in the data reading thread, data conversion thread, and data cleaning thread. Flow control is divided into three levels: low speed, medium speed, and high speed. It can be set dynamically according to the actual business scenario. The specific flow control is based on the current normal business scenario, when the data reading thread, data conversion thread, and data deletion thread work at the same time, and the number of written data files, read data files, and deleted data files can be calculated to meet the minimum obligations under normal circumstances.

[0085] This application is based on the video cloud distributed system architecture. The client first writes video recordings, pictures and other data files to multiple storage nodes in the hot data storage pool in the storage space through the data writing thread in a multi-copy mode. The data of each copy is completely consistent. As time goes on, the space capacity utilization rate of the system is very low. At the same time, the probability of data reading and the value of data are decreasing. Therefore, in order to ensure data readability and value, the erasure code selection strategy is dynamically set to determine the target erasure code rule of the initial data file. This application can select different erasure code strategies based on the rule configuration instruction for different life cycles, that is, to generate each erasure code rule based on the rule configuration instruction. When the data type of the initial data file is multi-copy data, the copy data is converted into the target erasure code data corresponding to the target erasure code rule, so that the initial data file can be divided into multiple blocks and written to different storage nodes. In other application scenarios, if the expiration time of the initial data file reaches the maximum expiration time, the target erasure code data is written to the node pool that exceeds the capacity, and the node pool that exceeds the capacity is used to store the target data file obtained after the format conversion of the initial data file that exceeds the maximum expiration time in the cold data storage pool. It is understandable that if the data reaches the maximum expiration time and you still want to ensure that the data is searchable, you can store the data in a node pool that exceeds the capacity according to the maximum redundancy rule. This not only ensures data security and readability, but also improves the overall storage space utilization of the system.

[0086] Multi-copy data can be in a distributed system architecture. For very important video recordings, pictures and other data, the data is saved as multiple identical data copies and written to multiple storage nodes in a multi-copy mode. In a distributed storage system, the MDS component is responsible for managing the metadata of the file system, including the structure, attributes, location and other information of the file. The MDS component can be used to process the metadata requests of the client and convert these requests into operations on the storage node. The MDS component can be an MDS service. The client interacts with the MDS component to create files, apply for objects, open files, return file ids, storage node ids, and is responsible for interacting with the system scheduling thread and the data deletion thread of the storage node and other related operations. The write data thread is responsible for receiving the write data request sent by the client and writing the received data to the current storage node. The system scheduling thread is responsible for loading the configuration file to set the expiration time and EC redundancy rules, analyzing the expired files on the MDS component side, and sending the expired file information, storage location information, and EC information to the read data thread of the storage node, which is responsible for interacting with the read data thread and data conversion thread of the storage node. The EC redundancy rules are the erasure code rules generated by the rule configuration instructions mentioned above. The data reading thread can read data from the storage node according to the data mode and verify the consistency of the data. The data conversion thread can be responsible for writing the data read by the data reading thread to multiple storage nodes according to the set EC mode, that is, writing the metadata read by the data reading thread to the target storage node according to the target erasure code rule. In some application scenarios, under normal circumstances, video cloud storage has business. In order to prevent the read and write traffic of the data reading thread and the data conversion thread from affecting the normal business, it is necessary to control the flow of the read and write process, that is, the flow control module controls the flow of the data reading thread, the data conversion thread, and the data cleaning thread. The data cleaning thread can be used to convert multiple copies of data between the initial data file and the target data file into erasure code data, or after the erasure code data is converted into erasure code data, that is, after the initial data file is successfully written, the initial data file corresponding to the metadata is deleted so that the target data file is used as metadata. The hot data storage pool has a faster storage speed and better disk performance. Real-time business data is written to this hot data storage pool. The data in the hot data storage pool is of high value and has a high probability of reading. The cold data storage pool has a large storage space, relatively poor disk performance, relatively slow read and write speed, low cost, and no real-time business. The cold data storage pool can be used to store data that has been expired for a long time. The data in the cold data storage pool has relatively low value and low read probability.

[0087] When the MDS service starts, the system scheduling thread is initialized, and the system scheduling thread loads the configuration file. The configuration file is mainly used to set the default expiration time of the file, the expiration time interval, and the EC erasure code rules. Different expiration times correspond to different EC erasure code rules, and the expiration time, EC erasure code rules and other information are updated to the database. The configuration file mainly contains the following configuration items. These configuration items can be loaded dynamically without restarting the service. The database is used to store the above-mentioned rule configuration information related to each erasure code rule. Please refer to Table 1, which can be an example table of rule configuration information.

[0088] Table 1: Example table of rule configuration information

[0089]

[0090] As shown in Table 1, the number of replicas can be replicas_num, which can indicate the number of storage nodes in the hot data storage pool in the storage space that the data writing thread writes in multi-copy mode. The local storage feedback result can be storage_local. If the local storage feedback result is 1, it indicates that the target erasure code data needs to be written to at least one of the storage nodes where the initial data file is located. The life cycle can be expired_internal, and the life cycle is used to indicate that the expiration time decreases every same preset number of days, and the preset number of days can be 15 days. The erasure code rule type can be ec_type, which can indicate the number of data blocks and check blocks in the erasure code rule corresponding to the erasure code rule type.

[0091] After the MDS service is successfully started, the storage node registers with the MDS service and reports its capacity. The client first creates a data file to the MDS service. The MDS service selects available storage nodes from the hot data storage pool based on each storage node's load, CPU, bandwidth and other performance parameters, and sends the fileId of the data file, which is the file identifier of the data file, the location of the data file storage node, the number of copies and other information to the client. After receiving the response, the client writes the same data block to multiple storage nodes according to the number of copies configured in Table 1. For example, the default number of copies used in this application is 3.

[0092] See also Figure 3 , Figure 3 It is a schematic diagram of the process of writing multiple copies of data in an embodiment of the data downgrade storage method of the present application.

[0093] like Figure 3 As shown, the client first writes data files such as video recordings and pictures to multiple storage nodes in the hot data storage pool in the storage space through a data writing thread in a multi-copy mode, and the data in each copy is completely consistent. Figure 3Each file block in the data storage pool is a data file, and the corresponding copy data of each data file is written to multiple storage nodes in the hot data storage pool. For example, block 1 corresponds to a data file, and the corresponding copy data of the data file corresponding to block 1 is written to the target storage node in the hot data storage pool, and the target storage nodes are node 1, node 2, and node 3.

[0094] After the client writes multiple copies of data to different storage nodes in the hot data storage pool, the storage node reports each copy of the data information to the MDS component. The MDS component can periodically start a scheduled task to calculate all data files that have exceeded the default expiration date and use the data file as the initial data file. For example, the default expiration time can be 0 days or 15 days. If it exceeds 15 days, the multiple copies of data are converted into erasure coded data. Exemplarily, the present application takes the erasure code rules of 4+1 and 29+1 as examples. If the file creation time is greater than 15 days and less than 30 days, the target erasure code rule is to convert the multiple copies of data into 4+1 erasure coded data. If the file creation time exceeds 30 days, the target erasure code rule is converted from 4+1 erasure code data to 29+1 erasure code data, that is, 29 data blocks and one redundant check block, and the currently expired data file and the corresponding file identifier, the target erasure code rule are placed in the data set. The data information set can be file_ec_map, the structure key of the data information set is the file identifier, and the value is the conversion mode between the initial data file and the target data file included in the target erasure code rule. Among them, the conversion mode can convert multiple copies of data into erasure code data or convert erasure code data into erasure code data.

[0095] The system scheduling thread can run regularly. For example, the system scheduling thread can be in the early morning when the business volume is small. The thread consumes the data information corresponding to each file identifier from the data information set. The system scheduling thread sends the target erasure code rule of the current data file to the MDS service. The MDS service determines that if the target erasure code rule is 4+1, it selects 5 storage nodes (or disks) with low system load and small network bandwidth from the hot data storage pool for the target erasure code data storage. If the target erasure code rule is 29+1, 30 storage nodes (or disks) with low system load and small network bandwidth are selected from the cold data storage pool nodes for the target erasure code data storage and returned to the system scheduling thread. The system scheduling thread sends the data information set to the data reading thread of one of the storage nodes with the lowest load. The data information set includes the file identifier, the initial erasure code rule, the target erasure code rule, whether the local storage feedback result corresponding to the converted target erasure code data is written first, the storage node information, the copy information, and other information. Among them, the file identifier is the file identifier of the initial data file, that is, the file fileId. The storage node information is used to indicate the target storage node location corresponding to the target erasure coded data. The replica information is used to indicate which nodes in the hot data storage pool the multiple replicas of the data are stored on if the data type of the initial data file is multi-copy data. The initial erasure code rule is used to indicate the erasure code rule corresponding to the initial data file. If the file creation time is greater than 15 days and less than 30 days, the initial erasure code rule of the initial data file is none, and the target erasure code rule is 4+1. If the file creation time is more than 30 days, the initial erasure code rule of the initial data file is 4+1, and the target erasure code rule of the initial data file is 29+1. Whether write priority is given to local in the local storage feedback result corresponding to the converted target erasure coded data can refer to whether one of the block data is written to the current storage node when the MDS component converts the data into EC erasure code to reduce network interaction. The data information set also includes the storage node location of the target erasure coded data block.

[0096] The storage node data reading thread receives the file identifier, target erasure code rule, whether to store local and erasure code data block storage node information sent by the system scheduling thread. The data reading thread preferentially determines whether the data type of the initial data file is multi-copy data. In response to the data type of the initial data file being a multi-copy data type, the erasure code rule corresponding to the multi-copy data type is determined as the target erasure code rule. In response to the data type of the initial data file being an erasure code data type, the erasure code rule corresponding to the erasure code data type is determined as the target erasure code rule. The data conversion thread obtains the target erasure code data corresponding to the initial data file according to the conversion strategy corresponding to the target erasure code rule. After the initial data file is successfully written, the data conversion thread sends the file identifier of the target data file to the system scheduling thread. The system scheduling thread sends the file identifier of the target data file to the MDS component. The MDS component updates the metadata information based on the file identifier of the target data file. The system scheduling thread is notified that the metadata update is completed. At this time, the file identifier and other information corresponding to the initial data file are placed in the queue to be deleted, which is consumed by the data cleaning thread to delete the initial data file.

[0097] See also Figure 4 , Figure 4 It is a schematic diagram of the process of converting a multi-copy data type into an erasure code data type in one embodiment of the data downgrade storage method of the present application.

[0098] In some application scenarios, if the data reading thread determines that the data type of the initial data file is a multi-copy data type, it reads a local copy of the data on the storage node of the initial data file and transfers the read data, the storage location where the data is to be written, whether it is stored locally (storage_local), the target erasure code rule and other information to the data conversion thread. Figure 4 As shown in the figure, the number of copies of the multi-copy data is 3, which are stored in nodes 1 to 3 of the hot data storage pool. DN represents a node. The data conversion thread receives the data transferred by the data reading thread, and converts it according to the target erasure code rule to obtain the target erasure code data, where the target erasure code rule is represented by a number of data blocks and a number of check blocks as N+M. Figure 4 As shown in the figure, N+M in the target erasure code rule is 4+1. Data is divided into stripes, each stripe contains N+M pieces, where N is the data block piece and M is the check data piece. According to the location information of the data to be written, the N+M pieces are written to different disks of different pre-allocated storage nodes. The pre-allocated different storage nodes are the target storage nodes obtained according to the above selection method. Figure 4As shown, the target storage nodes corresponding to the target erasure coded data are respectively in nodes 4 to 8 of the hot data storage pool. Each storage node returns a response whether the current piece is written successfully. When the data conversion thread receives a successful response from each storage node to write the piece, that is, the result of the successful writing of the initial data file, the file identifier of the target data file is submitted to the system scheduling thread, notifying the system scheduling thread that the initial data file corresponding to the target data file can be deleted. When the system scheduling thread receives the fileId, it first saves the metadata of the initial data file. After the storage node reports the latest block information of the file, that is, after the storage node reports the location information of the target storage node where the target data file of the file is located, the MDS component updates the file metadata information and notifies the system scheduling thread that the metadata update is complete. At this time, the file identifier and other information corresponding to the initial data file are placed in the queue to be deleted, which is consumed by the data cleaning thread to delete the initial data file.

[0099] See also Figure 5 , Figure 5 It is a schematic diagram of the process of converting the erasure code data type into the erasure code data type in an embodiment of the data degradation storage method of the present application.

[0100] In some application scenarios, if the data reading thread determines that the data type of the initial data file is an erasure code data type, the data block is read according to the initial erasure code rule of the initial data file. For example, in the initial erasure code rule, several data blocks and several check blocks are represented as A+B, such as Figure 5 As shown, in the initial erasure code rule, A+B is 4+1. Read data blocks from A storage nodes respectively. If there is a data block reading failure, continue to read B check blocks to ensure the original data is read successfully. The read data, the storage location where the data is to be written, whether it is stored locally (storage_local), the target erasure code rule and other information are transferred to the data conversion thread. The data conversion thread receives the data transferred by the data reading thread, and converts it according to the target erasure code rule to obtain the target erasure code data, where the target erasure code rule, a number of data blocks and a number of check blocks are represented as N+M. As shown in Figure 5As shown, N+M in the target erasure code rule is 29+1. Data is divided into stripes, each stripe contains N+M pieces, where N is the data block piece and M is the check data piece. According to the location information of the data to be written, the N+M pieces are written to different disks of different pre-assigned storage nodes. The pre-assigned different storage nodes are the target storage nodes obtained according to the above selection method. Each storage node returns a response whether the current piece is written successfully. When the data conversion thread receives a successful response from each storage node to write a piece, that is, the result of the successful writing of the initial data file, the file identifier of the target data file is submitted to the system scheduling thread to notify the system scheduling thread that the initial data file corresponding to the target data file can be deleted. When the system scheduling thread receives the fileId, it first saves the metadata of the initial data file. After the storage node reports the latest block information of the file, that is, after the storage node reports the location information of the target storage node where the target data file of the file is located, the MDS component updates the file metadata information and notifies the system scheduling thread that the metadata update is complete. At this time, the file identifier and other information corresponding to the initial data file are placed in the queue to be deleted, which is consumed by the data cleaning thread to delete the initial data file.

[0101] In other application scenarios, putting the file identifier and other information corresponding to the initial data file into the queue to be deleted can be to find the initial file data corresponding to the file identifier corresponding to the initial data file, and put the initial data file into the queue to be deleted as the data to be deleted. The data to be deleted in the queue to be deleted in this application may include multiple copies of data and erasure code data, which exist in the form of files on the storage node, and the file name is obtained by converting the file identifier, that is, the file fileId into hexadecimal. The serial number after the file name is different, so the original deletion is indiscriminate deletion. The data cleaning thread defaults to sending 50 files for deletion per heartbeat. The specific number of files to be deleted per heartbeat can be set in the configuration file. The data cleaning thread sends the file identifier and other information of the file to be deleted to the corresponding storage node. After receiving the file, the storage node converts the file identifier into the file name and deletes it directly. However, in order to avoid the storage node deleting files and occupying a large amount of CPU and other resources, the storage node file deletion thread will also limit the speed of deleted files.

[0102] Exemplarily, through the above process, within 15 days of the creation of the data file, the data file is stored in the storage space in the form of multiple copies of data. When the data file expires for 15 days, the data file is converted from multiple copies of data to erasure code rules according to the current corresponding target erasure code rules. At this time, the number of data blocks and the number of check blocks contained in the target erasure code rules are 4+1 respectively, and the data file is in the hot data storage pool at this time. As time goes by, the data value and read probability are both decreasing. When the data file expires for 30 days, the data file is converted from the initial erasure code rule 4+1 to the target erasure code rule 29+1 according to the current corresponding target erasure code rule. At this time, the file is stored in the cold data storage pool. This downgraded storage method not only ensures data integrity and consistency, but also reduces the reading and storage pressure of a single node, and improves the utilization rate of system sorting space. It can be understood that the present application does not limit the number of erasure code rules, nor does it limit the number of data blocks and check blocks in the erasure code rules. As time goes by, the total number of data blocks and check blocks in the target erasure coding rule determined for the data file is greater than the total number of data blocks and check blocks in the initial erasure coding rule of the initial data file.

[0103] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0104] See also Figure 6 , Figure 6 It is a structural diagram of an embodiment of a data degradation storage device of the present application. The data degradation storage device 60 includes an acquisition module 61, a determination module 62 and a conversion module 63; the acquisition module 61 is used to acquire the expiration time of the initial data file in the storage space, and the initial data file is a stored data file that exceeds the default expiration period; the determination module 62 is used to determine the erasure code rule corresponding to the target preset time period as the target erasure code rule of the initial data file in response to the expiration time of the initial data file being within the target preset time period, and the target preset time period is one of several preset time periods, and different preset time periods correspond to different erasure code rules; the conversion module 63 is used to convert the format of the initial data file according to the target erasure code rule to obtain the target data file, and the data volume of the target data file is less than the data volume of the initial data file.

[0105] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0106] For the functions performed by each module, please refer to the data downgrade storage method, which will not be repeated here.

[0107] See also Figure 7 , Figure 7 : is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72, and the processor 72 is used to execute the program instructions stored in the memory 71 to implement the steps in the above data degradation storage method embodiment. In a specific implementation scenario, the electronic device 70 may include but is not limited to: a multi-camera device, a microcomputer, and a server. In addition, the electronic device 70 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0108] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in the above-mentioned data degradation storage method embodiment. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0109] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0110] See also Figure 8 , Figure 8 The computer-readable storage medium 80 stores program instructions 801, which implement the steps in any of the above data degradation storage method embodiments when executed by a processor.

[0111] In the above scheme, in response to the expiration time of the initial data file in the acquired storage space being within the target preset time period, the erasure code rule corresponding to the target preset time period is determined as the target erasure code rule of the initial data file, wherein the initial data file that exceeds the default expiration period is determined to the target preset time period according to the relationship between the expiration time and a number of preset times to determine a more suitable target erasure code rule, and the format of the initial data file is converted according to the target erasure code rule to obtain a target data file, the data volume of the target data file is smaller than the data volume of the initial data file, and thus the target data file occupies less storage space than the initial data file, thereby releasing the memory of the storage space.

[0112] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0113] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0114] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A data downgrade storage method, characterized in that: The method comprises: Obtaining an expiration time of an initial data file in the storage space, wherein the initial data file is a stored data file that exceeds a default expiration time; In response to the expiration time of the initial data file being within a target preset time period, determining the erasure coding rule corresponding to the target preset time period as the target erasure coding rule of the initial data file, wherein the target preset time period is one of several preset time periods, and different preset time periods correspond to different erasure coding rules; The format of the initial data file is converted according to the target erasure code rule to obtain a target data file, wherein the data volume of the target data file is smaller than the data volume of the initial data file.

2. The method according to claim 1, characterized in that The storage space includes a hot data storage pool and a cold data storage pool, and the format conversion of the initial data file according to the target erasure code rule to obtain the target data file includes: Based on the data type of the initial data file, determining a storage pool into which the target data file should be written, the storage pool being the hot data storage pool or the cold data storage pool; Converting the initial data file into target erasure code data using the target erasure code rule; The target erasure coded data is written into the storage pool to obtain a target data file.

3. The method according to claim 2, characterized in that The storage pool includes a plurality of candidate storage nodes. Before writing the target erasure coded data into the storage pool to obtain the target data file, the method further includes: Selecting a target number of storage nodes from the plurality of candidate storage nodes as target storage nodes, wherein the target number is obtained by the target erasure coding rule; Using the location of the storage pool where the target storage node is located as the target storage location of the initial data file; Writing the target erasure coded data into the storage pool to obtain a target data file includes: The target erasure coded data is written into the target storage location to obtain a target data file.

4. The method according to claim 2, characterized in that: The method further comprises: In response to the storage pool being the hot data storage pool; Dividing the hot data storage pool to obtain a cache storage area; Writing the target erasure coded data into the storage pool to obtain a target data file includes: The target erasure coded data is written into a cache storage area in the hot data storage pool to obtain a target data file.

5. The method according to any one of claims 1 to 4, characterized in that: The data type of the initial data file includes a multi-copy data type or an erasure code data type, the plurality of preset time periods include a first type of preset time period related to the multi-copy data type and a second type of preset time period related to the erasure code data type, the first type of preset time period is before the second type of preset time period, and in response to the expiration time of the initial data file being within a target preset time period, determining the erasure code rule corresponding to the target preset time period as the target erasure code rule of the initial data file comprises: In response to the target preset time period being the first type of preset time period, determining the erasure coding rule corresponding to the multi-copy data type as the target erasure coding rule; or, In response to the target preset time period being the second-type preset time period, the erasure coding rule corresponding to the erasure coding data type is determined as the target erasure coding rule.

6. The method according to any one of claims 1 to 4, characterized in that: After converting the format of the initial data file according to the target erasure code rule to obtain the target data file, the method further includes: Determining whether the initial data file is successfully written; In response to the initial data file being successfully written, the initial data file is deleted.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtaining a flow control instruction for a target thread, wherein the flow control instruction is used to set the number of files processed by the target thread each time, wherein the target thread includes a data reading thread, a data conversion thread and / or a data cleaning thread, wherein the data reading thread is used to obtain the initial data file, the data conversion thread is used to perform format conversion on the initial data file, and the data cleaning thread is used to delete the initial data file; Flow control processing is performed on the target thread based on the flow control instruction.

8. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Obtain a rule configuration instruction, where the rule configuration instruction is used to indicate rule configuration information of each of the erasure code rules, where each of the rule configuration information includes at least one of a preset time period corresponding to each of the erasure code rules, a data type of an initial data file corresponding to each of the erasure code rules, and a number of storage nodes required for each of the erasure code rules; Based on the rule configuration instruction, each of the erasure code rules is generated.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 8.

Images