Data storage method, device, electronic device and storage medium

Abstract

The present disclosure provides a data storage method, device, electronic device and storage medium, which relate to the field of database technology, and in particular to the field of data storage technology. The specific implementation scheme is: split the data to be stored into N data blocks; encode the N data blocks to obtain N+M first copies; write each first copy of the N+M first copies into N+M first storage spaces respectively, and store one first copy in each first storage space; when the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first copies in the X first storage spaces to X second storage spaces, and recycle the X first storage spaces; wherein N and M are positive integers, and X is a positive integer less than or equal to M. The data storage method proposed in the present disclosure can facilitate data reading and writing.

Description

Technical Field

[0001] The present disclosure relates to the field of database technology, and in particular to the field of data storage technology. Background Art

[0002] In the data storage field, hybrid storage solutions are sometimes used, whereby a storage system utilizes two or more different storage media. A typical hybrid storage solution involves deploying two storage clusters, each using a different type of storage media. All data is then divided into two mutually exclusive sets, each stored in one storage cluster. Because data is stored in different storage clusters, reading and writing data requires first determining the cluster, which can be inconvenient. Summary of the Invention

[0003] The present disclosure provides a data storage method, apparatus, device, and storage medium.

[0004] According to one aspect of the present disclosure, a data storage method is provided, comprising:

[0005] Split the data to be stored into N data blocks;

[0006] Encoding the N data blocks to obtain N+M first copies;

[0007] Writing each of the N+M first replicas into N+M first storage spaces respectively, with each first storage space storing one first replica;

[0008] When the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to X second storage spaces, and reclaim the X first storage spaces;

[0009] Wherein, N and M are positive integers, and X is a positive integer less than or equal to M.

[0010] According to another aspect of the present disclosure, there is provided a data storage device, comprising:

[0011] an encoding module, configured to split the data to be stored into N data blocks; and encode the N data blocks to obtain N+M first copies;

[0012] a writing module, configured to write each of the N+M first replicas into the N+M first storage spaces respectively, with each first storage space storing one first replica;

[0013] a migration module, configured to, when the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to the X second storage spaces, and reclaim the X first storage spaces;

[0014] Wherein, N and M are positive integers, and X is a positive integer less than or equal to M.

[0015] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method of any embodiment of the present disclosure.

[0019] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the method according to any embodiment of the present disclosure when executed by a processor.

[0021] The data storage method proposed in the present disclosure can facilitate data reading.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0024] Figure 1 It is a schematic diagram of a hybrid storage method;

[0025] Figure 2 This is a schematic diagram of an application scenario of the present disclosure;

[0026] Figure 3 is a flowchart of an implementation of a data storage method 300 according to an embodiment of the present disclosure;

[0027] Figure 4is a schematic diagram of data storage using a data storage method according to an embodiment of the present disclosure;

[0028] Figure 5 is a structural diagram of a data storage device 500 according to an embodiment of the present disclosure;

[0029] Figure 6 is a structural diagram of a data storage device 600 according to another embodiment of the present disclosure;

[0030] Figure 7 is a structural block diagram of an electronic device 700 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Hybrid storage refers to the use of two or more different storage media within a storage system. For example, the common approach to implementing hybrid storage using two different storage media is to deploy two storage clusters, each using a different type of storage medium. All data is then divided into two mutually exclusive sets, each stored in the two storage clusters. Figure 1 This is a diagram of a hybrid storage method. Figure 1 As shown in the figure, two clusters are deployed in the storage system, Cluster A and Cluster B. Cluster A and Cluster B use storage medium A (referred to as Medium A) and storage medium B (referred to as Medium B), respectively. Data is split into two mutually exclusive sets by upstream services according to certain rules and stored in two isolated clusters. When writing data, the data must first be determined in which cluster it should be stored before executing the write operation. When reading data, the data must also be determined in which cluster before executing the read operation. This makes data reading and writing inconvenient and inefficient.

[0033] Figure 2 This is a schematic diagram of an application scenario of the present disclosure. Figure 2As shown, the application scenario may include a terminal 210 and a data storage server 220. The terminal 210 and the data storage server 220 are connected via a network. The terminal 210 can interact and transmit data with the data storage server 220. The terminal 210 can upload data to be stored to the data storage server 220, and the data storage server 220 stores the data. The terminal 210 can also send a download request to the data storage server 220, requesting to download the data stored in the data storage server 220. The data storage server 220 can transmit the specified data to the terminal 210 according to the download request.

[0034] The data storage server 220 may be a server cluster, or any server in the server cluster, or any functional module in the server, or any storage space in the server, which is not limited in the embodiment of the present invention. The terminal may be a mobile terminal or a computer terminal.

[0035] The present disclosure proposes a data storage method that can store different copies in different storage media based on the multi-copy storage architecture of a key-value (KV) storage system, thereby realizing a hybrid storage system based on two heterogeneous storage media in one system.

[0036] The data storage method proposed in this disclosure can be based on redundant protection technology within data storage technology. A simple redundant protection method involves replicating a piece of data into multiple identical copies, each stored on a different set of storage nodes. For example, data A can be replicated into three copies, stored on nodes 1, 3, and 4, randomly selected from the entire cluster. Multiple copies provide significant data protection. For example, if a node fails and data is lost, data A is still stored on the other nodes.

[0037] Currently, there are various coding technologies that implement data redundancy protection. Taking erasure code (EC) as an example, EC coding can achieve data redundancy protection by calculating and storing redundant blocks.

[0038] For example, when writing data to a distributed storage system, the data is divided into N data blocks (N is a positive integer, typically even). The EC encoding algorithm then calculates M redundant blocks (M is a positive integer, such as 2, 3, or 4). Data blocks and redundant blocks are collectively referred to as data replicas, and each replica (N+M replicas in total) is stored on different nodes. EC encoding technology only requires N replicas to restore the original data. Therefore, as long as no more than M nodes are damaged, data loss is guaranteed.

[0039] The space utilization of EC encoding is N / (N+M), and the reliability of the data is determined by the value of M. Data reliability is negatively correlated with space utilization. For example, the larger M is, the higher the reliability is, but the space utilization is correspondingly lower.

[0040] Taking 8-12EC encoding as an example, 8-12EC encoding splits the data to be stored into 8 data blocks, then calculates 4 redundant blocks using EC encoding, resulting in 12 copies (including data blocks and redundant blocks). These 12 copies are stored on 12 servers (such as storage space), thus achieving redundant protection of the data.

[0041] Based on redundancy protection and related coding technologies, the present disclosure proposes a data storage method. Figure 3 The following is a flowchart of a data storage method according to an embodiment of the present disclosure, including:

[0042] S301: Split the data to be stored into N data blocks;

[0043] S302: Encode N data blocks to obtain N+M first copies;

[0044] S303: Writing each of the N+M first replicas into N+M first storage spaces respectively, with each first storage space storing one first replica;

[0045] S304: When the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to X second storage spaces, and reclaim the X first storage spaces;

[0046] Wherein, N and M are positive integers, and X is a positive integer less than or equal to M.

[0047] In some implementations, the N represents the number of data blocks corresponding to the encoding, and the M represents the number of redundancies corresponding to the encoding.

[0048] In an embodiment of the present disclosure, the first storage space and the second storage space may belong to different storage clusters or servers, respectively. Since the copy of the data is first written to the first storage space, there is no need to determine which cluster the data should be written to when writing the data, which facilitates data writing; and after migrating X first copies to the second storage space, there are still at least N first copies stored in the first storage space (N+MX≥N), so when reading data under normal circumstances, it can be read directly from the first storage space, which facilitates data reading. It can be seen that the data storage method proposed in the present disclosure can facilitate data reading and writing.

[0049] It should be noted that the "first" in the above "first copy" is only used to distinguish it from the "second copy" that will appear later. "First" and "second" are not used to indicate importance or order, etc. "First copy" and "second copy" both refer to copies of data.

[0050] In some embodiments, the first storage space and the second storage space may utilize different storage media. For example, if the first storage space utilizes a disk (e.g., a hard disk drive (HDD)) as the storage medium, the first storage space may be a disk storage space; and if the second storage space utilizes a magnetic tape as the storage medium, the second storage space may be a magnetic tape storage space. Because magnetic tapes are less expensive than magnetic disks, hybrid magnetic and magnetic tape storage can reduce storage server costs.

[0051] Figure 4 This is a schematic diagram of a data storage method using an embodiment of the present disclosure. Figure 4 In the example, the data to be stored is split into 8 data blocks, and 8-12EC encoding is performed on the 8 data blocks to obtain 4 redundant blocks. The 8 data blocks and 4 redundant blocks together make up 12 replicas (replicas can be data blocks or redundant blocks). The 12 replicas are written into 12 first storage spaces (such as disk storage spaces). Disk storage space is a pre-configured space with a fixed size (such as 8G, 16G, or 32G, etc.). Disk storage space can be called a disk volume or volume. One data block may not be enough to fill a disk storage space. The replicas obtained by splitting and encoding other data to be stored can continue to be written into the disk storage space.

[0052] When a disk storage space is full, the replicas in part of the disk storage space can be migrated to the secondary storage space, and this part of the disk storage space can be reclaimed. For example, if the 12 disk storage spaces mentioned above are full, a portion (e.g., X) of the disk storage spaces can be selected from the 12 disk storage spaces, and the replicas in the selected X disk storage spaces can be migrated as a whole to X secondary storage spaces, and the X disk storage spaces can be reclaimed. After the recycling, the disk storage space is freed and can be rewritten with data.

[0053] In some implementations, the second storage space may be a tape storage space.

[0054] Regarding the selection of disk storage spaces for data migration, that is, how to determine the value of X, the present disclosure may consider the following points:

[0055] First, when the first storage space is a disk storage space and the second storage space is a tape storage space, since the latency of reading data from the tape is greater than that from the disk, to ensure low latency for data reading, at least N data blocks should be retained in the first storage space (i.e., the disk storage space). This way, if the first storage space is intact, only the replicas in the first storage space need to be read to restore the original data. Therefore, X should be less than or equal to M to ensure that the number of replicas stored in the first storage space is greater than or equal to N (i.e., N + M X ≥ N).

[0056] Second, since failures in the first storage space are inevitable, a certain amount of redundant data can be stored in the first storage space. For example, if the original data is split into N copies, one copy can be stored in each of the N+MX first storage spaces, for a total of N+MX copies, of which MX are redundant copies.

[0057] Under the above premise, there may be two situations when reading data:

[0058] Case 1: The number of failed first storage spaces is no more than MX. In this case, since the number of remaining first storage spaces is no less than N, the original data can be restored by reading the replicas in the first storage space without reading the second storage space.

[0059] Case 2: The number of failed first storage spaces is greater than MX. In this case, since the number of remaining first storage spaces is less than N, in addition to reading the replicas in the first storage space, it is also necessary to read some of the replicas in the second storage space to restore the original data.

[0060] If the second storage space uses a magnetic tape as the storage medium, the data reading delay of the magnetic tape is relatively long, which will lead to a relatively long reading delay in the above situation 2. Therefore, the lower the probability of the above situation 2 occurring, the better.

[0061] Based on the above analysis, the value of X can be determined according to at least one of the following:

[0062] the quantity of the first storage space included in the storage system;

[0063] Failure probability of the first storage space;

[0064] The mean time to repair of the first storage space;

[0065] The fault tolerance of the storage system.

[0066] Specifically, based on the number of first storage spaces included in the storage system, the failure probability of the first storage spaces, the mean time to repair of the first storage spaces, and various values ​​of X, the probability of scenario 2 occurring for various values ​​of X is determined. A determination is then made as to whether the probability of scenario 2 occurring exceeds the failure tolerance of the storage system. Ultimately, the value of X that maximizes the probability of scenario 2 occurring while maintaining the failure tolerance of the storage system is selected.

[0067] by Figure 4 Take the 8-12EC encoding shown as an example. Figure 4 In the example shown, the data is split into N (N = 8) data blocks, resulting in N + M (M = 4) replicas. X replicas are then written to tape storage. Assume the disk storage cluster consists of 5,000 machines (or, alternatively, the storage system contains 5,000 disk storage spaces), the probability of disk storage failure is 5% per year, and the average repair time for disk storage spaces is 1 day.

[0068] To determine the value of X, we can calculate the probability of situation 2 occurring when X takes different values, and determine whether the probability exceeds the fault tolerance of the storage system.

[0069] For example, when X = 1, one replica is stored in the tape storage space, and the remaining 11 replicas are stored in 11 disk storage spaces. If four or more (including four) of the 11 disk storage spaces are damaged, fewer than eight replicas are stored in the disk storage space, and some replicas must be read from the tape storage space to restore the original data, resulting in Situation 2 described above. Therefore, based on the number of first storage spaces included in the storage system (e.g., 5000), the failure probability of the first storage space (e.g., 5% per year), and the average failure repair time of the first storage space (e.g., 1 day), the probability of Situation 2 occurring can be calculated as:

[0070]

[0071] Therefore, the above situation 2 occurs once every 1 / 0.009=111 days on average.

[0072] When X = 2, two copies are stored in two tape storage spaces, and the remaining 10 copies are stored in 10 disk storage spaces. If three or more (including three) of the 10 disk storage spaces are damaged, fewer than eight copies are stored in the disk storage spaces, and some copies must be read from the tape storage spaces to restore the original data, resulting in Situation 2. Therefore, based on the number of first storage spaces included in the storage system (e.g., 5000), the failure probability of the first storage spaces (e.g., 5% per year), and the average failure repair time of the first storage spaces (e.g., 1 day), the probability of Situation 2 occurring can be calculated as:

[0073]

[0074] Therefore, the above situation 2 occurs once every 1 / 0.054=18.5 days on average.

[0075] For another example, when X = 3, three replicas are stored in three tape storage spaces, and the remaining nine replicas are stored in nine disk storage spaces. If two or more (including two) of the nine disk storage spaces are damaged, fewer than eight replicas are stored in the disk storage spaces, and some replicas must be read from the tape storage spaces to restore the original data, resulting in Situation 2. Therefore, based on the number of first storage spaces included in the storage system (e.g., 5000), the failure probability of the first storage spaces (e.g., 5% per year), and the average failure repair time of the first storage spaces (e.g., 1 day), the probability of Situation 2 occurring can be calculated as:

[0076]

[0077] Therefore, the above situation 2 occurs once every 1 / 0.235=4.2 days on average.

[0078] As can be seen from the above example, the larger the value of X, the greater the probability of the above situation 2 occurring. Assume that the calculated probabilities of the above situation 2 occurring when X=1 and X=2 do not exceed the fault tolerance of the storage system; since the cost of tape storage space is relatively low, from a cost perspective, more tape storage space can be used as much as possible, so X=2 is determined. When the value of X is 2, the tape needs to be read for data recovery approximately once every 18.5 days. It can be seen that the method of determining the above X proposed in the embodiment of the present disclosure can reduce the cost of the storage system while meeting the system fault tolerance.

[0079] The above embodiments describe a specific method for determining the value of X. Regarding data writing and space recovery after data deletion, the present disclosure may employ the following methods:

[0080] In some implementations, when the N+M first storage spaces are full, selecting X first storage spaces includes:

[0081] When the first storage space is full, the write operation of the first storage space is disabled.

[0082] In some implementations, when the N+M first storage spaces are full, selecting X first storage spaces includes:

[0083] The first storage spaces in the storage system are scanned, and when it is detected that the write operations of the N+M first storage spaces are disabled, X first storage spaces from the N+M first storage spaces are selected according to a predetermined rule.

[0084] Since magnetic tapes do not support random writing, the embodiment of the present disclosure adopts an offline asynchronous batch writing method. First, multiple copies are written to each disk storage space separately, and then X disk storage spaces are selected from them, and the copies in the X disk storage spaces are written offline asynchronously in batches to the tape storage space. Specifically, a copy of each data is filled in a disk storage space of a fixed size when writing. After the entire disk storage space is filled, the disk storage space is closed for writing operations. The system will scan offline to see which disk storage spaces are closed for writing operations and all copies are still on the disk, and then select X disk storage spaces according to certain rules, migrate the copies in the selected disk storage spaces to the tape storage space, and reclaim the corresponding space in the disk storage space. It can be seen that the data writing method in the embodiment of the present disclosure can adapt to the characteristics of different storage media and improve the efficiency of data writing.

[0085] For space reclamation after data deletion, the embodiments of the present disclosure may adopt different reclamation methods for disk storage space and tape storage space:

[0086] For example, the disk storage space is reclaimed in the following way: after deleting the copies in the disk storage space, the capacity occupied by the disk storage space is reduced, the free capacity of the disk storage space is reclaimed, and a new disk storage space is created using the reclaimed free capacity. The new disk storage space can be used for writing new data, while the original disk storage space is always in a write-closed state.

[0087] The method for recycling tape storage space is as follows: during the process of shrinking the disk storage space, the data in the corresponding tape storage space remains unchanged. As the deletion action proceeds, when there are many deleted copies in the disk storage space (for example, accounting for more than 50%), the copies in the corresponding tape storage space can be deleted as a whole to reclaim the corresponding tape storage space. For the remaining copies in the disk storage space, new copies can be re-encoded to obtain new copies, and then the new copies can be written to the new disk storage space; after the disk storage space is full, the copies in part of the disk storage space can be migrated to the tape storage space. It can be seen that the space recycling method proposed in the embodiment of the present disclosure can adapt to the characteristics of different storage media, improve recycling efficiency, and ensure the consistency of data stored in different storage media.

[0088] For example, in some embodiments, the data storage method proposed in the present disclosure may further include:

[0089] Deleting the first copy in the first storage space and reclaiming free space in the first storage space;

[0090] If the ratio of the reclaimed free space to the initial capacity of the first storage space is greater than or equal to a predetermined threshold, re-encode the remaining first replicas in the first storage space to obtain N+M second replicas, and delete the first replica in the second storage space corresponding to the first storage space; wherein the first replica stored in the second storage space corresponding to the first storage space and the first replica stored in the first storage space are obtained by encoding the same data block;

[0091] The N+M second copies are written into the N+M first storage spaces respectively.

[0092] In some embodiments, the data deletion process, i.e., the step of deleting the first copy of the first storage space, may be initiated when certain trigger conditions are met. The trigger conditions may include receiving a user's data deletion instruction, automatically deleting expired data, and the like.

[0093] For example, consider encoding the data to be stored using the 8-12EC encoding method and storing 10 replicas in disk storage spaces. Assuming each disk storage space or tape storage space is 8GB in size, 8GB replicas (which can be one or more replicas of the data to be stored) are written to each of the 12 disk storage spaces. Once the data is full, write operations to the disk storage spaces are disabled. Two of the disk storage spaces are then selected, and the replicas in these selected disk storage spaces are migrated to two tape storage spaces, which are then reclaimed. During data deletion, the replicas in the disk storage spaces are deleted in real time, and the free space created by the deleted replicas is reclaimed. When the ratio of free space to initial capacity in a disk storage space reaches a predetermined threshold (e.g., 50%), for example, when the free space in a disk storage space reaches 4GB, all replicas in the corresponding two tape storage spaces are deleted, and the two tape storage spaces are reclaimed. Furthermore, the remaining copies in the 10 disk storage spaces are re-encoded to generate a total of 12 new copies, each of which is 4G in size, and each copy is written to the 12 disk storage spaces respectively; after these disk storage spaces are full, the copies in two of the disk storage spaces can be migrated to the tape storage space. It can be seen that since the tape storage space does not have the ability to write or delete randomly, the data writing and deletion of the tape storage space in the embodiment of the present disclosure adopts an asynchronous batch method to facilitate the reading and writing of data. Moreover, since the cost of the tape storage space is relatively low, even if some copies in the tape storage space are not deleted in a timely manner, the cost of the storage server can still be saved.

[0094] The present disclosure also provides a data storage device. Figure 5 is a structural diagram of a data storage device according to an embodiment of the present disclosure, such as Figure 5 The data storage device 500 shown includes:

[0095] The encoding module 510 is configured to split the data to be stored into N data blocks; encode the N data blocks to obtain N+M first copies;

[0096] A writing module 520 is configured to write each of the N+M first replicas into the N+M first storage spaces, with each first storage space storing one first replica;

[0097] a migration module 530 configured to, when the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to the X second storage spaces, and reclaim the X first storage spaces;

[0098] Wherein, N and M are positive integers, and X is a positive integer less than or equal to M.

[0099] The present disclosure also provides another data storage device, Figure 6 is a structural diagram of a data storage device according to another embodiment of the present disclosure. Figure 6 As shown, the data storage device 600 includes:

[0100] The encoding module 510, the writing module 520, the migration module 530 and the recycling module 640, the writing module also includes a write operation control submodule 521 and a scanning and selection submodule 522, wherein the encoding module 510, the writing module 520, the migration module 530 and the corresponding modules mentioned above are the same and will not be repeated here.

[0101] In some possible implementations, N represents the number of data blocks corresponding to the encoding, and M represents the number of redundancies corresponding to the encoding.

[0102] In some possible implementations, the first storage space includes a disk storage space, and the second storage space includes a tape storage space.

[0103] In some possible implementations, the migration module 530 determines X according to at least one of the following:

[0104] the number of first storage spaces included in the storage system;

[0105] Failure probability of the first storage space;

[0106] The mean time to repair of the first storage space;

[0107] The fault tolerance of the storage system.

[0108] In some possible implementations,

[0109] The write operation control submodule 521 is configured to disable the write operation of the first storage space when the first storage space is full.

[0110] In some possible implementations, the scanning and selection submodule 522 is used to: scan each first storage space in the storage system, and when it is detected that the write operation of N+M first storage spaces is closed, select X first storage spaces from the N+M first storage spaces according to a predetermined rule.

[0111] In some possible implementations, the recovery module 640 is configured to:

[0112] The first copy in the first storage space is deleted, and the free space in the first storage space is reclaimed; when the ratio of the reclaimed free space to the initial capacity of the first storage space is greater than or equal to a predetermined threshold, the remaining first copies in the first storage space are re-encoded to obtain N+M second copies, and the first copy in the second storage space corresponding to the first storage space is deleted; wherein the first copy stored in the second storage space corresponding to the first storage space and the first copy stored in the first storage space are obtained by encoding the same data block; and the N+M second copies are respectively written into the N+M first storage spaces.

[0113] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0114] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0115] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0116] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0117] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0118] Various components in device 700 are connected to I / O interface 705, including an input unit 706, such as a keyboard, mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, optical disk, etc.; and a communication unit 709, such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0119] The computing unit 701 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the data storage method. For example, in some embodiments, the data storage method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the data storage method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the data storage method in any other appropriate manner (e.g., by means of firmware).

[0120] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0121] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0122] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0123] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0124] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0125] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0126] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0127] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data storage method, comprising: Split the data to be stored into N data blocks; Encoding the N data blocks to obtain N+M first copies; Writing each of the N+M first replicas into N+M first storage spaces respectively, with each first storage space storing one first replica; When the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to X second storage spaces, and reclaim the X first storage spaces; Wherein, N and M are positive integers, and X is a positive integer less than or equal to M; The final value of X is determined by: determining the probability of a specific situation occurring under various values ​​of X based on the number of first storage spaces included in the storage system, the failure probability of the first storage spaces, and the mean failure repair time of the first storage spaces, and determining whether the probability of the specific situation occurring exceeds the failure tolerance of the storage system, so as to select the maximum value of X for which the probability of the specific situation occurring does not exceed the failure tolerance of the storage system as the final value of X; wherein the specific situation is that the number of failed first storage spaces is greater than MX.

2. The method according to claim 1, wherein The N represents the number of data blocks corresponding to the encoding, and the M represents the number of redundancies corresponding to the encoding.

3. The method according to claim 1, wherein When the N+M first storage spaces are full, selecting X first storage spaces from the N+M first storage spaces includes: When the first storage space is full, the write operation of the first storage space is disabled.

4. The method according to claim 1, wherein When the N+M first storage spaces are full, selecting X first storage spaces from the N+M first storage spaces includes: The first storage spaces in the storage system are scanned, and when it is detected that the write operations of the N+M first storage spaces are disabled, X first storage spaces from the N+M first storage spaces are selected according to a predetermined rule.

5. The method according to any one of claims 1 to 4, further comprising: Deleting the first copy in the first storage space and reclaiming free space in the first storage space; If the ratio of the reclaimed free space to the initial capacity of the first storage space is greater than or equal to a predetermined threshold, re-encode the remaining first replicas in the first storage space to obtain N+M second replicas, and delete the first replica in the second storage space corresponding to the first storage space; wherein the first replica stored in the second storage space corresponding to the first storage space and the first replica stored in the first storage space are obtained by encoding the same data block; The N+M second copies are written into the N+M first storage spaces respectively.

6. The method according to claim 1, wherein The encoding includes an erasure code (EC) encoding.

7. The method according to claim 1, wherein The first storage space includes a disk storage space, and the second storage space includes a tape storage space.

8. A data storage device comprising: An encoding module, used for splitting the data to be stored into N data blocks; Encoding the N data blocks to obtain N+M first copies; a writing module, configured to write each of the N+M first replicas into the N+M first storage spaces respectively, with each first storage space storing one first replica; a migration module, configured to, when the N+M first storage spaces are full, select X first storage spaces from the N+M first storage spaces, migrate the first replicas in the X first storage spaces to the X second storage spaces, and reclaim the X first storage spaces; Wherein, N and M are positive integers, and X is a positive integer less than or equal to M; The migration module determines the final value of X in the following manner: Based on the number of first storage spaces included in the storage system, the failure probability of the first storage spaces, and the mean repair time of the first storage spaces, the probability of a specific situation occurring under various values ​​of X is determined, and a determination is made as to whether the probability of the specific situation occurring exceeds the failure tolerance of the storage system. The maximum value of X, for which the probability of the specific situation occurring does not exceed the failure tolerance of the storage system, is selected as the final value of X. The specific situation is that the number of failed first storage spaces is greater than MX.

9. The device according to claim 8, wherein The N represents the number of data blocks corresponding to the encoding, and the M represents the number of redundancies corresponding to the encoding.

10. The device according to claim 8, wherein The migration module includes: The write operation control submodule is configured to disable the write operation of the first storage space when the first storage space is full.

11. The device according to claim 8, wherein The migration module includes: The scanning and selecting submodule is used to scan each of the first storage spaces in the storage system, and when it is detected that the write operation of the N+M first storage spaces is closed, select X first storage spaces from the N+M first storage spaces according to a predetermined rule.

12. The apparatus according to any one of claims 8 to 11, further comprising: a recycling module, configured to delete the first copy in the first storage space and reclaim free space in the first storage space; When the ratio of the reclaimed free space to the initial capacity of the first storage space is greater than or equal to a predetermined threshold, the remaining first copies in the first storage space are re-encoded to obtain N+M second copies, and the first copy in the second storage space corresponding to the first storage space is deleted; wherein the first copy stored in the second storage space corresponding to the first storage space and the first copy stored in the first storage space are obtained by encoding the same data block; and the N+M second copies are written into the N+M first storage spaces respectively.

13. The device according to claim 8, wherein The encoding includes an erasure code (EC) encoding.

14. The device according to claim 8, wherein The first storage space includes a disk storage space, and the second storage space includes a tape storage space.

15. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images