Data storage mode conversion method and system

Abstract

The data storage mode conversion method provided by the present invention is applied to a data storage mode conversion system; the system includes: a data processing server, an encoding processing server, and an information management device; the method includes: using the data processing server to store original data in a backup storage stripe; when the backup storage stripe successfully stores data, using the information management device to set the state of the backup storage stripe to a first preset state; when the encoding processing server meets preset encoding conditions, using the encoding processing server to obtain target data stored in the target backup storage stripe and generate encoding data corresponding to the target data; and using the encoding processing server to store the target data and the encoding data in the encoding storage stripe. By separating the data processing server from the encoding processing server and separating the backup storage process from the data storage mode conversion process, the present invention reduces the overall load pressure on each server in the system.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and more particularly to a method and system for converting data storage modes. Background Art

[0002] In a distributed file system, after backing up data, encoded data is often also needed. In backup storage, the storage space ratio between original data and mirrored data is at least 1:1; in encoded storage, the ratio is often 2:1 or even higher. This shows that encoded storage can save more storage space than backup storage.

[0003] Currently, the process of converting data from backup storage to encoded storage typically involves a data processing server encoding the read data immediately after a backup storage stripe is full, and then storing the encoded data back in the encoded storage stripe. However, executing both the backup data storage and storage conversion processes in a compact, continuous manner on the same server, or even within the same process, inevitably leads to a significant increase in server memory consumption and I / O data volume. This can also cause server congestion during large-scale computations, leading to responsiveness issues in other service programs and disrupted business operations. It can also significantly consume the server's CPU resources, causing abnormalities in other business operations on the server.

[0004] Therefore, there is an urgent need for an effective data storage conversion solution to avoid excessive load pressure on the server during the data processing process. Summary of the Invention

[0005] In view of this, the present invention provides a data storage mode conversion method and system to solve the technical problem that the existing data storage mode conversion solution may cause excessive load pressure on the server and affect the normal operation of the server business.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A data storage mode conversion method is applied to a data storage mode conversion system; the system includes: a data processing server, a coding processing server and an information management device; the method includes:

[0008] The data processing server is used to store the original data in a backup storage stripe; the backup storage stripe includes a plurality of data block groups, wherein each data block group is used to store the original data, and the data stored in each data block group serves as a backup for each other;

[0009] When the backup storage stripe successfully stores data, using the information management device to set the state of the backup storage stripe to a first preset state;

[0010] When the encoding processing server meets the preset encoding condition, the encoding processing server is used to obtain target data stored in a target backup storage stripe and generate encoding data corresponding to the target data; the target backup storage stripe is any backup storage stripe having the first preset state; the target data is the original data stored in the target backup storage stripe;

[0011] The encoding processing server is used to store the target data and the encoded data in an encoded storage strip; the encoded storage strip includes original data blocks and encoded data blocks, the original data blocks are used to store the target data, and the encoded data blocks are used to store the encoded data.

[0012] Preferably, storing the original data in the backup storage stripe by using the data processing server includes:

[0013] Using the data processing server to store the original data in each data block group of the backup storage stripe;

[0014] Setting the health value of the data block group that successfully stores data in the backup storage stripe to a first preset value;

[0015] When the health value of at least one data block group in the backup storage stripe is a first preset value, it is determined that the backup storage stripe stores data successfully.

[0016] Preferably, the encoding processing server meets the preset encoding conditions including:

[0017] The idle rate of the encoding processing server is higher than a preset idle rate threshold;

[0018] and / or,

[0019] The current system time of the encoding processing server reaches the preset encoding time.

[0020] Preferably, when the encoding processing server meets the preset encoding condition, using the encoding processing server to obtain the target data stored in the target backup storage stripe and generating encoding data corresponding to the target data further includes:

[0021] When the encoding processing server meets the preset encoding condition, querying by the encoding processing server whether there is a backup storage stripe with the first preset state;

[0022] When it is found that there is a backup storage stripe in the first preset state, the encoding processing server is used to obtain target data stored in the target backup storage stripe and generate encoding data corresponding to the target data.

[0023] Preferably, before storing the target data and the encoded data in the encoded storage stripe using the encoding processing server, the method further includes:

[0024] Using the encoding processing server, applying for an encoding data stripe, wherein the encoding data stripe only includes the encoding data block;

[0025] Accordingly, storing the target data and the encoded data in the encoded storage stripe by using the encoding processing server includes:

[0026] storing, using the encoding processing server, the encoded data corresponding to the target data into the encoded data block;

[0027] The encoding processing server is used to take a data block of any data block group that successfully stores the target data in the backup storage stripe as the original data block, and splice it with the encoded data block to obtain an encoded storage stripe storing the target data and the encoded data.

[0028] Preferably, after obtaining the encoded storage strip storing the target data and the encoded data, the method further comprises:

[0029] The information management device is used to release resources of other data block groups in the backup storage stripe; wherein the other data block groups are data block groups in the backup storage stripe excluding the data block groups used for splicing the encoding storage stripe.

[0030] A data storage mode conversion system, comprising: a data processing server, a coding processing server and an information management device;

[0031] The data processing server is used to store the original data in a backup storage stripe; the backup storage stripe includes a plurality of data block groups, wherein each data block group is used to store the original data, and the data stored in each data block group is a backup of each other;

[0032] The information management device is configured to set the state of the backup storage stripe to a first preset state when the backup storage stripe successfully stores data;

[0033] The encoding processing server is configured to, when the encoding processing server meets a preset encoding condition, obtain target data stored in a target backup storage stripe and generate encoding data corresponding to the target data; the target backup storage stripe is any backup storage stripe having the first preset state; and the target data is original data stored in the target backup storage stripe;

[0034] The encoding processing server is further used to store the target data and the encoded data in an encoded storage strip; the encoded storage strip includes original data blocks and encoded data blocks, the original data blocks are used to store the target data, and the encoded data blocks are used to store the encoded data.

[0035] Preferably, the data processing server is specifically used to store the original data respectively in each data block group of the backup storage stripe; set the health value of the data block group that successfully stores data in the backup storage stripe to a first preset value; when the health value of at least one data block group in the backup storage stripe is the first preset value, it is determined that the backup storage stripe has successfully stored data.

[0036] Preferably, the encoding processing server meets the preset encoding conditions including:

[0037] The idle rate of the encoding processing server is higher than a preset idle rate threshold;

[0038] and / or,

[0039] The current system time of the encoding processing server reaches the preset encoding time.

[0040] Preferably, the encoding processing server is specifically used to:

[0041] When the encoding processing server meets the preset encoding conditions, it queries whether there is a backup storage stripe with the first preset state; when it is found that there is a backup storage stripe with the first preset state, the target data stored in the target backup storage stripe is obtained, and encoding data corresponding to the target data is generated.

[0042] Preferably, the encoding processing server is also used to apply for an encoded data stripe, which only includes the encoded data block; store the encoded data corresponding to the target data in the encoded data block; use the data block of any data block group that successfully stores the target data in the backup storage stripe as the original data block, and splice it with the encoded data block to obtain an encoded storage stripe that stores the target data and the encoded data.

[0043] Preferably, the information management device is further configured to, after obtaining the encoded storage stripe storing the target data and the encoded data, release resources of other data block groups in the backup storage stripe using the information management device;

[0044] The other data block groups are data block groups in the backup storage stripe excluding the data block groups used for splicing the encoding storage stripe.

[0045] It can be seen from the above technical solution that the data storage mode conversion method and system provided by the present invention are based on a data processing server and an encoding processing server that are separately set to execute the backup storage process and the data storage mode conversion process respectively, thereby reducing the data processing pressure of the data processing server; and, after completing the backup storage, the data processing server first sets the backup storage stripe to the first preset state, and no longer directly executes the encoding mode conversion work; then the encoding processing server decides the timing of executing the backup mode conversion according to the preset encoding conditions, so that the backup storage process and the data storage mode conversion process are independent and separated from each other, thereby releasing the tight coupling relationship between the backup storage process and the data storage mode conversion process, improving the flexibility of the data storage mode conversion, and reducing the data processing pressure of the encoding processing server, thereby alleviating the load pressure of each server in the entire system as a whole, avoiding excessive consumption of resources of each server, and avoiding negative impacts on other services of each server. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A flow chart of a data storage mode conversion method provided by an embodiment of the present invention;

[0048] Figure 2 Another flow chart of the data storage mode conversion method provided in an embodiment of the present invention;

[0049] Figure 3 Another flow chart of a data storage mode conversion method provided by an embodiment of the present invention;

[0050] Figure 4 A structural diagram of a data storage mode conversion system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] In a distributed file system, for data security reasons, a stripe typically consists of at least two data blocks. Whether using coded or backup storage, if a data block in a stripe is invalid and data cannot be read from it, data recovery can be performed using data from other valid data blocks. A valid data block means that data can be read from it.

[0053] Backup storage, also known as "backup system," means that a stripe consists of at least two data block groups, and the data stored in different data block groups is exactly the same, that is, they back up each other. The data block organization of the backup storage stripe can be shown in Table 1:

[0054] Table 1 Data block organization of backup storage stripe

[0055]

[0056] The backup storage stripe shown in Table 1 above includes three data block groups: data block group 1, data block group 2, and data block group 3. Each data block group has four data blocks arranged in data storage order. For example, data block group 1 includes data blocks 11, 12, 13, and 14, arranged in data storage order.

[0057] The data block organization of a backup storage stripe can be viewed as a data block matrix (as shown in Table 1). In this data block matrix, the data blocks are divided horizontally by data block groups and vertically by data storage. For example, horizontal data blocks 11, 12, 13, and 14 form data block group 1, while vertical data blocks 11, 21, and 31 store identical data. The data stored between different data blocks in the same data block group is continuous, associated data within the file and is non-duplicate data. For example, the data stored between data blocks 11, 12, 13, and 14 in data block group 1 is continuous data within the file and is non-duplicate data.

[0058] When storing data, the data processing server does not cache it. Instead, it stores the data directly in the stripe and stores it in the order of the data blocks in the backup storage stripe. For example, it stores data block *1 first, then data block *2, then data block *3, and finally data block *4. The "*" represents any group of data blocks. Storing data block *1 means storing data blocks 11, 21, and 31.

[0059] Encoded storage, also known as "encoded system," means that a stripe consists of a data block group, which includes raw data blocks for storing original data and coded data blocks for storing coded data. The number of coded data blocks is less than the number of raw data blocks.

[0060] When the data processing server has finished storing a backup storage stripe and needs to convert the backup system to a code system, it needs to allocate the corresponding memory space. The memory space allocation is calculated by multiplying the total storage space size of a data block group of the backup storage stripe by a preset multiple. This preset multiple can be 3 / 2, or other values ​​greater than 1. The data stored in the backup storage stripe is read into the allocated memory space. Taking the example of a data block group of the backup storage stripe having 4 data blocks and a preset multiple of 3 / 2, the allocated memory space is shown in Table 2 below:

[0061] Table 2 Memory space allocation

[0062]

[0063] The memory data block is used to cache the original data stored in the backup storage stripe, and the memory encoding block is used to cache the encoded data obtained after encoding the original data.

[0064] First, any data block that can be successfully read from data block *1 of the backup storage stripe is selected, and its data is read into memory data block 1; any data block that can be successfully read from data block *2 of the backup storage stripe is selected, and its data is read into memory data block 2, and so on, directly reading all the original data of the backup storage stripe into the memory data block.

[0065] Then, the data of memory data blocks 1 to 4 are encoded, and the encoded data obtained after encoding is stored in memory encoding block 1 and memory encoding block 2.

[0066] Then, memory data blocks 1 to 4 and memory coding blocks 1 to 2 are associated one by one with the original data blocks 1 to 4 and coded data blocks 1 to 2 of the coded storage stripe, and the corresponding data is stored in the data blocks of the coded storage stripe. The data block organization of the coded storage stripe can be shown in Table 3 below:

[0067] Table 3 Data block organization of coded storage stripe

[0068]

[0069] Finally, the original backup storage stripe is replaced with the obtained coded storage stripe. The relevant information of the original backup storage stripe is set to the initial value for subsequent data writing, and the information of the coded storage stripe is recorded in the corresponding file association table of the information management device.

[0070] In backup storage, one copy of original data requires at least one corresponding copy of mirror data, meaning the storage ratio of original data to mirror data is 1:n (n≥1). Encoded storage is designed to reduce the space occupied by backup storage and lower hardware costs.

[0071] The encoding system generally uses a 2:1 storage ratio, meaning the storage ratio of original data to encoded data is 2:1. Compared to the 1:1 storage ratio used by the backup system, the encoding system only occupies 2 / 3 of the storage space of the backup system, thus reducing space utilization and saving hardware costs. Of course, to further save storage space, the encoding system can also use a storage ratio of 3:1, 4:1, or even larger multiples. However, a larger ratio reduces security, so a 2:1 ratio is generally the best setting.

[0072] In a distributed file system, a server's IO, CPU, memory and other resources are limited. In the traditional backup-to-code conversion solution, a data processing server immediately executes the backup-to-code conversion process after a backup storage stripe is full. The entire process is carried out continuously, which puts excessive load pressure on the data processing server and even affects the normal operation of other services of the data processing server.

[0073] In this regard, the present invention provides a data storage mode conversion method and system. The data storage mode conversion method is applied to the data storage mode conversion system, which at least includes a data processing server, an encoding processing server and an information management device.

[0074] See also Figure 1 , Figure 1 A flow chart of a data storage mode conversion method provided by an embodiment of the present invention.

[0075] like Figure 1 As shown, the data storage mode conversion method of this embodiment includes:

[0076] S110: Utilize the data processing server to store the original data into a backup storage stripe.

[0077] A backup storage stripe, also known as a "backup stripe," utilizes a backup storage method. A backup storage stripe includes multiple data block groups, each of which stores the original data. The data stored in each data block group serves as a backup of each other. For details on the data block organization of a backup storage stripe, see Table 1.

[0078] Raw data refers to the original, unencoded data that the user needs to write into a file.

[0079] The data processing server stores the original data in the backup storage stripe. If the backup storage stripe stores the data successfully (the stripe is full, or storage stops due to file closure), the data processing server reports to the information management device that the stripe stores the data successfully.

[0080] S120: When the backup storage stripe successfully stores data, the information management device is used to set the state of the backup storage stripe to a first preset state.

[0081] When the backup storage stripe successfully stores data, it also indicates that the backup storage stripe has completed the accurate conversion from the backup system to the encoding system. Therefore, the first preset state can be used to indicate not only that the backup storage stripe has successfully stored data, but also that the backup storage stripe is in a state awaiting encoding. Furthermore, the first preset state can be a state in which data storage is successful, or a state in which encoding is pending.

[0082] When the backup storage stripe fails to store data, the information management device can also be used to set the state of the backup storage stripe to a second preset state. The second preset state is different from the first preset state and can specifically be a data storage failure state or an unencodable state.

[0083] S130: When the encoding processing server meets the preset encoding condition, the encoding processing server is used to obtain target data stored in the target backup storage stripe, and generate encoding data corresponding to the target data.

[0084] The target backup storage stripe is any backup storage stripe in the first preset state. As the data processing server continuously stores raw data in the backup storage stripe, at least one backup storage stripe in the first preset state may be obtained, that is, at least one backup storage stripe to be encoded may be obtained.

[0085] When the encoding processing server meets the preset encoding conditions, a backup storage stripe is selected from at least one backup storage stripe to be encoded as a target backup storage stripe, and then the encoding process is performed. The target data is the original data stored in the target backup storage stripe.

[0086] The traditional solution for converting backup storage to coded storage is that the same data processing server performs both the backup storage process and the data storage conversion process. Moreover, the backup storage process and the data storage conversion process are continuous, tightly coupled business processes, which puts enormous load pressure on the data processing server.

[0087] In the present invention, the backup storage process is still executed by the data processing server, but the data storage mode conversion process (i.e., the process of converting the backup system to the encoding system) is handed over to the additional encoding processing server for execution. The additional encoding processing server is used to share the data processing pressure of the data processing server and reduce the load of the data processing server.

[0088] Moreover, the present invention further separates and makes independent the backup storage process and the data storage mode conversion process based on the data processing server and the encoding processing server that are set separately from each other. Specifically, after the data processing server stores the original data in the backup storage stripe, it first suspends the backup storage stripe, that is, enters the waiting encoding state, instead of directly executing the encoding process; and when the encoding processing server is ready to execute the backup mode to encoding mode work is determined by the encoding processing server based on whether it meets the preset encoding conditions, thereby releasing the tight coupling relationship between the backup storage process and the data storage mode conversion process, making the backup storage process and the data storage mode conversion process no longer a continuous and uninterrupted business process, and eliminating the need to immediately execute the encoding system conversion project after the backup storage, thereby reducing the data processing pressure of the encoding processing server.

[0089] In a distributed storage server cluster, multiple encoding processing servers can be set up to further alleviate the system load pressure.

[0090] S140: Utilize the encoding processing server to store the target data and the encoded data into an encoded storage stripe.

[0091] The coded storage stripe includes original data blocks and coded data blocks, the original data blocks are used to store the target data, and the coded data blocks are used to store the coded data.

[0092] The data storage mode conversion method provided in this embodiment is based on a data processing server and an encoding processing server that are separately set up to respectively execute the backup storage process and the data storage mode conversion process, thereby reducing the data processing pressure of the data processing server; and, after completing the backup storage, the data processing server first sets the backup storage stripe to the first preset state, and no longer directly performs the encoding mode conversion work; then the encoding processing server decides the timing of executing the backup mode conversion according to the preset encoding conditions, so that the backup storage process and the data storage mode conversion process are independent and separated from each other, thereby releasing the tight coupling relationship between the backup storage process and the data storage mode conversion process, improving the flexibility of the data storage mode conversion, and reducing the data processing pressure of the encoding processing server, thereby alleviating the load pressure of each server in the entire system as a whole, avoiding excessive consumption of resources of each server, and avoiding negative impacts on other services of each server.

[0093] See also Figure 2 , Figure 2 Another flow chart of the data storage mode conversion method provided by an embodiment of the present invention.

[0094] like Figure 2 As shown, the data storage mode conversion method of this embodiment includes:

[0095] S210: Using the data processing server to store the original data in each data block group of the backup storage stripe.

[0096] S220: Setting the health value of the data block group in which data is successfully stored in the backup storage stripe to a first preset value.

[0097] The first preset value represents health, that is, the data is valid or available.

[0098] Accordingly, when a data block group in the backup storage stripe fails to store data, the health value of the data block group in the backup storage stripe that fails to store data can be set to a second preset value, wherein the second preset value indicates unhealthy, that is, invalid or unusable data.

[0099] S230: When the health value of at least one data block group in the backup storage stripe is a first preset value, determine that the backup storage stripe stores data successfully.

[0100] When the health value of at least one data block group in the backup storage stripe is the first preset value, it indicates that the complete original data can be read from the backup storage stripe, and it can be determined that the backup storage stripe has successfully stored the data; if the health values ​​of all data block groups in the backup storage stripe are the second preset value, it indicates that the complete original data cannot be read from the backup storage stripe, and it can be determined that the backup storage stripe has failed to store the data.

[0101] S240: When the backup storage stripe successfully stores data, the information management device is used to set the state of the backup storage stripe to a first preset state.

[0102] S250: When the encoding processing server meets the preset encoding condition, the encoding processing server is used to query whether there is a backup storage stripe with the first preset state.

[0103] In one example, the encoding processing server meeting the preset encoding condition includes: an idle rate of the encoding processing server being higher than a preset idle rate threshold; and / or a current system time of the encoding processing server reaching a preset encoding time.

[0104] The preset idle rate threshold may be 50%, 80%, or 30%, and may be flexibly set according to specific needs.

[0105] The preset encoding time may be a pre-set time or a preset encoding time determined according to a preset encoding period. For example, the preset encoding period may be 10 seconds, that is, a query request is made every 10 seconds.

[0106] The encoding processing server is used to query whether a backup storage stripe currently has the first preset state. Specifically, the encoding processing server can query an information management device to determine whether a backup storage stripe currently has the first preset state. The information management device stores relevant information about all stripes.

[0107] S260: When it is found that a backup storage stripe having the first preset state currently exists, the encoding processing server is used to obtain target data stored in the target backup storage stripe, and generate encoding data corresponding to the target data.

[0108] The target backup storage stripe is any backup storage stripe in the first preset state; and the target data is the original data stored in the target backup storage stripe.

[0109] When executing the conversion process from backup system to encoding system, the information management device can also mark the backup storage stripe as being in the conversion state, which is used to prohibit operations on the backup storage stripe during the conversion process to avoid conversion failure due to the conversion process being affected.

[0110] S270: Utilize the encoding processing server to store the target data and the encoded data into an encoded storage stripe.

[0111] The coded storage stripe includes original data blocks and coded data blocks, the original data blocks are used to store the target data, and the coded data blocks are used to store the coded data.

[0112] The data storage mode conversion method provided in this embodiment, when the idle rate of the encoding processing server is higher than a preset idle rate threshold, and / or the current system time of the encoding processing server reaches a preset encoding time, the encoding processing server is used to query the backup storage stripe in the first preset state, and the backup mode to encoding mode conversion process is performed for the backup storage stripe in the first preset state that is queried, thereby further reducing the load pressure of the encoding processing server.

[0113] See also Figure 3 , Figure 3 Another flow chart of the data storage mode conversion method provided by an embodiment of the present invention.

[0114] like Figure 3 As shown, the data storage mode conversion method of this embodiment includes:

[0115] S310: Utilize the data processing server to store the original data into a backup storage stripe.

[0116] S320: When the backup storage stripe successfully stores data, the information management device is used to set the state of the backup storage stripe to a first preset state.

[0117] S330: When the encoding processing server meets the preset encoding condition, the encoding processing server is used to obtain target data stored in the target backup storage stripe, and generate encoding data corresponding to the target data.

[0118] The target backup storage stripe is any backup storage stripe in the first preset state; the target data is the original data stored in the target backup storage stripe.

[0119] S340: Utilize the encoding processing server to apply for an encoding data stripe, where the encoding data stripe only includes encoding data blocks.

[0120] The coded data strips may be as shown in Table 4 below:

[0121] Table 4 Encoded data strip

[0122] Encoded data block 1 Encoded data block 2

[0123] S350: Using the encoding processing server, store the encoded data corresponding to the target data into the encoded data block.

[0124] S360: Using the encoding processing server, a data block of any data block group that successfully stores the target data in the backup storage stripe is used as an original data block, and is spliced ​​with the encoded data block to obtain an encoded storage stripe storing the target data and the encoded data.

[0125] Wherein, any data block group that successfully stores the target data in the backup storage stripe is used as the original data block, which can be specifically shown in the following Table 5:

[0126] Table 5 Original data block

[0127] Original data block 1 Original data block 2 Original data block 3 Original data block 4

[0128] By splicing the original data block in Table 5 with the coded data block in Table 4, an encoded storage strip storing target data and coded data can be obtained, as shown in Table 6 below:

[0129] Table 6 Encoded Storage Strip

[0130]

[0131] It can be seen that the encoded storage stripe includes original data blocks and encoded data blocks, the original data blocks are used to store the target data, and the encoded data blocks are used to store the encoded data.

[0132] S370: Utilize the information management device to release resources of other data block groups in the backup storage stripe.

[0133] The other data block groups are data block groups in the backup storage stripe excluding the data block groups used for splicing the encoding storage stripe.

[0134] A set of readable original data blocks is selected from the backup storage stripe and combined with the encoded data blocks of the encoded data stripe to obtain a new stripe, namely the encoded storage stripe. Then, the other unused data blocks in the backup storage stripe are released for use when creating a new stripe.

[0135] The data storage mode conversion method provided by this embodiment uses the encoding processing server to apply for an encoded data stripe that only includes encoded data blocks, rather than directly applying for an encoded storage stripe that includes both original data blocks and encoded data blocks, thereby reducing the resource occupation of storage space during the data storage mode conversion process; then, a data block of any data block group that successfully stores the target data in the backup storage stripe is used as the original data block, and is spliced ​​with the encoded data block to obtain an encoded storage stripe that stores the target data and the encoded data, thereby improving the utilization rate of the data blocks in the backup storage stripe; and after the encoded storage stripe is spliced, the resources of other data block groups in the backup storage stripe are released for use when creating a new stripe, thereby further improving the utilization rate of the storage space.

[0136] Taking a specific application scenario as an example, the data storage mode conversion system mainly includes: a data processing server, an encoding processing server, and an information management device. In addition, the system may also include: a disk service device.

[0137] When a user needs to write data to a file, the original data to be written to the file is first transmitted to the data processing server, which then requests a backup storage stripe from the information management device. This embodiment uses a backup storage stripe with three data block groups, where the data stored in different data block groups is the same.

[0138] According to the storage ratio of encoded storage, when the number of data blocks in a data block group of a backup storage stripe is at least 2, the number of encoded data blocks in the encoded storage stripe is at least 1. This embodiment is configured according to a storage ratio of 2:1 for original data to encoded data. Of course, in other embodiments, a 3:1 or 4:1 ratio can also be used. However, the higher the storage ratio, the lower the security. Generally, to ensure relatively good security, a 2:1 or 3:1 ratio is recommended. This embodiment uses a 2:1 ratio for description.

[0139] Assume that the disk corresponding to a data block can store 4MB of data, and a stripe consists of three data block groups, each with four data blocks. Therefore, a stripe can store 16MB of data. This 16MB of data is stored simultaneously in three data block groups, with the data stored in different data block groups being identical. That is, data block group 1, data block group 2, and data block group 3 all store the same 16MB of data, with identical content and storage order.

[0140] Each data block is associated with a disk service device link address and data block number. The link address indicates which disk the data block belongs to, and the data block number indicates the data block's number on the disk. The data block number can be used to calculate the physical offset of the data block on the corresponding disk service device. For example, the corresponding physical address can be obtained by multiplying the data block size by the data block number.

[0141] After requesting a backup storage stripe from the information management device, the data processing server links the data blocks in the backup storage stripe to the corresponding disk service device. It then sequentially writes the received raw data to the data blocks in the backup storage stripe. Specifically, the first 4MB of data is written to the disk corresponding to data block *1, the next 4MB of data is written to the disk corresponding to data block *2, the third 4MB of data is written to the disk corresponding to data block *3, and the fourth 4MB of data is written to the disk corresponding to data block *4. If the user's initial raw data transfer does not fill the data blocks of the backup storage stripe—that is, if the backup storage stripe's 16MB of data space is not full—then the user must continue writing new raw data to the backup storage stripe through the data processing server, continuing to write new data after the physical offset of the last written data. For example, if 5MB of data was previously written to the backup storage stripe, then upon receiving the new 4MB of data, writing the next 4MB of data will begin at the end of the 5MB of data. Similarly, if a data block was not completely written last time, the next time the data block is written, it will continue writing from the end of the previous data block until the previous data block is full, and then write the following data blocks. If the second time does not fill a stripe, it will need to be written a third time, a fourth time, and so on, until a stripe is full or the file is closed and no more writing is done.

[0142] If a file is closed and a backup storage stripe is not full, the file is completely written. Even if the last stripe is not full, the remaining space in the backup storage stripe must be filled with zeros, indicating that the remaining data in the file needs to be cleared to zeros. Suppose that when the file is closed, only 5MB of data has been written to the backup storage stripe. This means that data block 1 is full and data block 2 has only 1MB of data. Then, the remaining 3MB of data block 2 must be cleared to zeros, and data blocks 3 and 4 must also be cleared to zeros before being written to the corresponding disks.

[0143] When writing data to a backup storage stripe, a health value is required. The health value is used to indicate the health of the backup storage stripe after writing data, and also indicates the health of the data in the stripe. The health value can be set separately for each data block group, as shown in Table 7 below:

[0144] Table 7 Health value of data block group

[0145] Group Health Data block group 1 Health 1 Data block group 2 Health 2 Data block group 3 Health 3

[0146] As can be seen in Table 7 above, each data block group has a corresponding health value. In this embodiment, the health value is either 1 or 0. If any data block in a data block group fails to store data, the health of the data block group is set to 0 (i.e., the second preset value). If all data blocks in the data block group successfully store data, the health value of the data block group is set to 1 (i.e., the first preset value). In the initial state where no data is written, the health value of the data block group is initially set to 1.

[0147] Assume that during the data storage process of data block group 1, if a data block encounters a network problem, a disk disconnection, or other problems, resulting in data storage failure, the health value of data block group 1 will be set to 0. During the data storage process of a data block group, as long as the health value of the data block group is detected to be 0, the subsequent data storage process will not be executed.

[0148] The data processing server reports the backup storage stripe that successfully stores data to the information management device. The information management device marks the backup storage stripe as in use or pending encoding and records the stripe ID of the backup storage stripe in the stripe list associated with the user file. When a backup storage stripe is first allocated, it is marked as in use.

[0149] When the encoding processing server is busy, it continues its ongoing encoding process and does not handle new encoding work. When the encoding processing server is idle, it queries the information management device at a preset time interval whether there is a backup system to encoding system conversion transaction to be processed, for example, once every 10 seconds. When converting the backup system data space to the encoding system data space, it is generally not so urgent to convert the data space immediately. Therefore, the timeliness of the conversion from the backup system to the encoding system is not a high requirement. The encoding processing server only needs to handle the new storage mode conversion at regular intervals.

[0150] The encoding processing server queries the information management device to determine whether the backup storage stripe is in use or awaiting encoding. If so, it retrieves all relevant information about the backup storage stripe from the information management device, including the stripe's individual data block groups, the link address of the disk service device corresponding to each data block, and the data block's data number. The encoding processing server then requests the encoded data blocks for the encoded storage stripe from the information management device (as shown in Table 4).

[0151] After obtaining the backup storage stripes and encoded data blocks, the encoding processing server allocates four memory data blocks and two memory encoding blocks. Assuming each data block requires 4MB of data space, the corresponding memory space is also 4MB. Six 4MB memory blocks are allocated, corresponding to original data block 1, original data block 2, original data block 3, original data block 4, encoded data block 1, and encoded data block 2, respectively.

[0152] Select a data block group with a health value of 1 in the backup storage stripe, link the disk service device corresponding to each data block in this data block group, and then read the data of this data block group into the memory data block. For example: Assume that the health value of data block group 1 of the backup storage stripe is 1, link it to the corresponding disk service device, and read the data of the original data block into the corresponding memory data block in the memory. If the reading of a certain original data block fails, it is necessary to read the data from the corresponding original data block of other data block groups into the memory. Assuming that the reading of data block 12 in data block group 1 fails, the data is read into the memory through data block 22 in data block group 2 with a health value of 1. Among them, the mapping relationship between the original data block and the memory data block can be shown in Table 8 below:

[0153] Table 8 Mapping relationship between original data block and original memory block

[0154] Original data block 1 Original data block 2 Original data block 3 Original data block 4 Memory data block 1 Memory data block 2 Memory data block 3 Memory data block 4

[0155] As shown in Table 8 above, original data block 1 is read into original memory block 1; original data block 2 is read into original memory block 2, and so on. The data in memory blocks 1 through 4 are then encoded, and the resulting encoded data is stored in memory encoding blocks 1 and 2. After encoding is complete, the corresponding disk service device is connected, and the encoded data is then stored on the disk corresponding to the encoded data block.

[0156] A data block group with a health value of 1 is selected from the backup storage stripe, and its data blocks are used as the original data blocks of the encoded storage stripe. In this embodiment, the four original data blocks in a data block group of the backup storage stripe are used as the original data blocks of the encoded storage stripe. Assuming that data block group 1 of the backup storage stripe is selected as the original data blocks of the encoded storage stripe, then the two encoded data blocks allocated from the information management device are the encoded data blocks of the encoded storage stripe. Then, the original data blocks are spliced ​​with the encoded data blocks to obtain a complete encoded storage stripe. When splicing the encoded storage stripes, the data storage order of all data blocks remains unchanged. The original data block 1, original data block 2, original data block 3, original data block 4, encoded data block 1, and encoded data block 2 are spliced ​​into a complete encoded storage stripe according to the storage order.

[0157] The information about each data block in the coded storage stripe (such as disk link information, disk block number, and data block order) is then reported to the information management device. The information management device creates a correlation table for these data blocks and stores them. It also marks the coded storage stripe as used, sets the health value of the coded storage stripe to 1, and generates a corresponding coded storage stripe ID.

[0158] The stripe ID of the backup storage stripe in the file information table is replaced with the stripe ID of the coded storage stripe. Before the backup storage stripe of the file is destroyed, the stripe described by the file and the data in the corresponding stripe can be operated.

[0159] Since the backup storage stripe previously had three data block groups, and one of the data block groups with a health value of 1 has now been separated and given to the encoding storage stripe, the backup storage stripe now has two data block groups remaining. The data in these two data block groups is no longer needed, so the information about the remaining data blocks of the backup storage stripe (such as the remaining eight data blocks and the backup stripe ID) is sent to the information management device. Assuming that we use four data blocks from data block group 1 of the backup storage stripe to splice the encoding storage stripe, then the data blocks of data block groups 2 and 3 remain.

[0160] Finally, the information management device reclaims the remaining eight data blocks in the backup storage stripe, placing them in its free data block pool and simultaneously clearing the backup storage stripe. The reclaimed data blocks are later made available for use in other files, enabling data block reuse.

[0161] The embodiments of the present invention also provide a data storage mode conversion method and system, which improves the throughput of the data processing server for user data, reduces the delay of the data processing server caused by data congestion, and flexibly increases the management and configuration of the server, making the resource utilization between servers more efficient.

[0162] An embodiment of the present invention also provides a data storage mode conversion system, which is used to implement the data storage mode conversion method provided by an embodiment of the present invention. The technical content of the data storage mode conversion system described below can be referenced in correspondence with the technical content of the data storage mode conversion method described above.

[0163] See also Figure 4 , Figure 4 A structural diagram of a data storage mode conversion system provided by an embodiment of the present invention.

[0164] like Figure 4 As shown, the data storage mode conversion system at least includes: a data processing server 10, an encoding processing server 20 and an information management device 30.

[0165] The data processing server 10 is used to store the original data in a backup storage stripe.

[0166] The backup storage stripe includes a plurality of data block groups, wherein each data block group is used to store the original data, and the data stored in each data block group serves as a backup for each other.

[0167] The information management device 30 is configured to set the state of the backup storage stripe to a first preset state when the backup storage stripe successfully stores data.

[0168] The encoding processing server 20 is configured to, when the encoding processing server meets a preset encoding condition, obtain target data stored in a target backup storage stripe and generate encoding data corresponding to the target data.

[0169] The target backup storage stripe is any backup storage stripe in the first preset state; the target data is the original data stored in the target backup storage stripe.

[0170] The encoding processing server 20 is further configured to store the target data and the encoded data in an encoded storage stripe.

[0171] The coded storage stripe includes original data blocks and coded data blocks, the original data blocks are used to store the target data, and the coded data blocks are used to store the coded data.

[0172] The data storage mode conversion system provided in this embodiment is based on a data processing server and an encoding processing server that are separately set up to respectively execute the backup storage process and the data storage mode conversion process, thereby reducing the data processing pressure of the data processing server; and, after completing the backup storage, the data processing server first sets the backup storage stripe to the first preset state, and no longer directly performs the encoding mode conversion work; then the encoding processing server decides the timing of executing the backup mode conversion according to the preset encoding conditions, so that the backup storage process and the data storage mode conversion process are independent and separated from each other, thereby releasing the tight coupling relationship between the backup storage process and the data storage mode conversion process, improving the flexibility of the data storage mode conversion, and reducing the data processing pressure of the encoding processing server, thereby alleviating the load pressure of each server in the entire system as a whole, avoiding excessive consumption of resources of each server, and avoiding negative impacts on other services of each server.

[0173] In other embodiments, the functions of the data processing server 10, the encoding processing server 20 and the information management device 30 in the data storage mode conversion system may be further described.

[0174] Among them, the data processing server 10 is specifically used to store the original data in each data block group of the backup storage stripe respectively; set the health value of the data block group that successfully stores data in the backup storage stripe to a first preset value; when the health value of at least one data block group in the backup storage stripe is the first preset value, it is determined that the backup storage stripe has successfully stored data.

[0175] The encoding processing server meeting the preset encoding condition includes: the idle rate of the encoding processing server is higher than a preset idle rate threshold; and / or the current system time of the encoding processing server reaches a preset encoding time.

[0176] Among them, the encoding processing server 20 is specifically used to: when the encoding processing server meets the preset encoding conditions, query whether there is a backup storage stripe with the first preset state; when it is queried that there is a backup storage stripe with the first preset state, obtain the target data stored in the target backup storage stripe, and generate encoding data corresponding to the target data.

[0177] Among them, the encoding processing server 20 is also used to apply for an encoded data stripe, which only includes the encoded data block; store the encoded data corresponding to the target data in the encoded data block; use the data block of any data block group that successfully stores the target data in the backup storage stripe as the original data block, and splice it with the encoded data block to obtain an encoded storage stripe that stores the target data and the encoded data.

[0178] The information management device 30 is further configured to, after obtaining the encoded storage stripe storing the target data and the encoded data, release resources of other data block groups in the backup storage stripe using the information management device. The other data block groups are data block groups in the backup storage stripe excluding the data block groups used to splice the encoded storage stripe.

[0179] The data storage mode conversion system provided in this embodiment, when the idle rate of the encoding processing server is higher than a preset idle rate threshold and / or the current system time of the encoding processing server reaches a preset encoding time, the encoding processing server is used to query the backup storage stripe in the first preset state, and the backup mode to encoding mode conversion process is performed for the queried backup storage stripe in the first preset state, thereby further reducing the load pressure of the encoding processing server.

[0180] Furthermore, the encoding processing server is used to apply for an encoded data stripe including only encoded data blocks, rather than directly applying for an encoded storage stripe including both original data blocks and encoded data blocks, thereby reducing the resource occupation of storage space during the data storage mode conversion process; then, a data block of any data block group that successfully stores the target data in the backup storage stripe is used as the original data block, and is spliced ​​with the encoded data block to obtain an encoded storage stripe storing the target data and the encoded data, thereby improving the utilization rate of the data blocks in the backup storage stripe; and after the encoded storage stripe is spliced, the resources of other data block groups in the backup storage stripe are released for use when creating a new stripe, thereby further improving the utilization rate of the storage space.

[0181] Finally, it should be noted that, in this document, relational terms such as first and first are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0182] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Based on this understanding, all or part of the contribution of the technical solution of the present application to the background technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present application.

[0183] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0184] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data storage mode conversion method, characterized in that: Applicable to a data storage mode conversion system; the system includes: a data processing server, a coding processing server and an information management device; the method includes: The data processing server is used to store the original data in a backup storage stripe; the backup storage stripe includes a plurality of data block groups, wherein each data block group is used to store the original data, and the data stored in each data block group serves as a backup for each other; When the backup storage stripe successfully stores data, using the information management device to set the state of the backup storage stripe to a first preset state; When the encoding processing server meets the preset encoding condition, the encoding processing server is used to obtain target data stored in a target backup storage stripe and generate encoding data corresponding to the target data; the target backup storage stripe is any backup storage stripe having the first preset state; the target data is the original data stored in the target backup storage stripe; Using the encoding processing server, applying for an encoding data stripe, wherein the encoding data stripe only includes encoding data blocks; storing, using the encoding processing server, the encoded data corresponding to the target data into the encoded data block; Using the encoding processing server, taking a data block of any data block group that successfully stores the target data in the backup storage stripe as an original data block, and splicing it with the encoded data block to obtain an encoded storage stripe storing the target data and the encoded data; The information management device is used to release resources of other data block groups in the backup storage stripe; wherein the other data block groups are data block groups in the backup storage stripe excluding the data block groups used for splicing the encoding storage stripe.

2. The method according to claim 1, wherein The storing of the original data into the backup storage stripe by using the data processing server includes: Using the data processing server to store the original data in each data block group of the backup storage stripe; Setting the health value of the data block group that successfully stores data in the backup storage stripe to a first preset value; When the health value of at least one data block group in the backup storage stripe is a first preset value, it is determined that the backup storage stripe stores data successfully.

3. The method according to claim 1, wherein The encoding processing server meets the preset encoding conditions including: The idle rate of the encoding processing server is higher than a preset idle rate threshold; and / or, The current system time of the encoding processing server reaches the preset encoding time.

4. The method according to claim 1, wherein When the encoding processing server meets the preset encoding condition, using the encoding processing server to obtain target data stored in the target backup storage stripe and generating encoding data corresponding to the target data further includes: When the encoding processing server meets the preset encoding condition, querying by the encoding processing server whether there is a backup storage stripe with the first preset state; When it is found that there is a backup storage stripe in the first preset state, the encoding processing server is used to obtain target data stored in the target backup storage stripe and generate encoding data corresponding to the target data.

5. A data storage mode conversion system, characterized in that: The system includes: a data processing server, a coding processing server and an information management device; The data processing server is used to store the original data in a backup storage stripe; the backup storage stripe includes a plurality of data block groups, wherein each data block group is used to store the original data, and the data stored in each data block group is a backup of each other; The information management device is configured to set the state of the backup storage stripe to a first preset state when the backup storage stripe successfully stores data; The encoding processing server is configured to, when the encoding processing server meets a preset encoding condition, obtain target data stored in a target backup storage stripe and generate encoding data corresponding to the target data; the target backup storage stripe is any backup storage stripe having the first preset state; and the target data is original data stored in the target backup storage stripe; The encoding processing server is further configured to apply for an encoded data stripe, the encoded data stripe comprising only encoded data blocks; store the encoded data corresponding to the target data in the encoded data blocks; and use a data block of any data block group that successfully stores the target data in the backup storage stripe as an original data block, and concatenate it with the encoded data block to obtain an encoded storage stripe storing the target data and the encoded data; The information management device is further configured to release resources of other data block groups in the backup storage stripe after obtaining the encoded storage stripe storing the target data and the encoded data; The other data block groups are data block groups in the backup storage stripe excluding the data block groups used for splicing the encoding storage stripe.

6. The system according to claim 5, wherein: The data processing server is specifically used to store the original data in each data block group of the backup storage stripe respectively; set the health value of the data block group that successfully stores data in the backup storage stripe to a first preset value; when the health value of at least one data block group in the backup storage stripe is the first preset value, it is determined that the backup storage stripe has successfully stored data.

7. The system according to claim 5, wherein: The encoding processing server meets the preset encoding conditions including: The idle rate of the encoding processing server is higher than a preset idle rate threshold; and / or, The current system time of the encoding processing server reaches the preset encoding time.

8. The system according to claim 5, wherein: The encoding processing server is specifically used for: When the encoding processing server meets the preset encoding conditions, it queries whether there is a backup storage stripe with the first preset state; when it is found that there is a backup storage stripe with the first preset state, the target data stored in the target backup storage stripe is obtained, and encoding data corresponding to the target data is generated.

Images