Data storage method, single-node server and device

Abstract

The present disclosure provides a data storage method, a single-node server and a device, which not only solves the strong dependence of single-node disaster recovery storage on Raid card hardware, but also improves the multi-level disaster recovery capability of data storage. The method includes: obtaining storage parameters of at least one single-node server, wherein the storage parameters represent the parameters used when slicing and encoding the original data to be stored; according to the storage parameters of the single-node server, determining N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with the M data blocks among the N first data blocks and the M second data blocks except the any N data blocks; and storing the data blocks corresponding to the original data in different disks of the single-node server respectively.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data storage and disaster recovery, and in particular to a data storage method, a single-node server and a device. Background Art

[0002] Faced with various possible disasters, enterprises need to conveniently and flexibly synchronize data residing in different databases based on heterogeneous environments. Therefore, they need to build a local and remote disaster recovery system that can resist or resolve various situations.

[0003] In the application environment of audio and video data storage of traditional single-node servers, it is usually necessary to rely on professional RAID (Redundant Array of Independent Disk) hardware resources to provide multi-level disaster recovery capabilities, such as RAID5 or RAID6. However, traditional hardware RAID solutions require sacrificing hardware resources to provide high-availability disaster recovery capabilities, which makes it difficult to simultaneously meet the needs of reducing hardware resource costs and improving disaster recovery capabilities. Summary of the invention

[0004] The present invention provides a data storage method, a single-node server and a device, which do not rely on professional RAID card hardware resources, and implement a multi-level disaster recovery storage solution in the form of software, which not only solves the strong dependence of single-node disaster recovery storage on Raid card hardware, but also improves the multi-level disaster recovery capability of data storage.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for storing data, the method comprising:

[0006] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0007] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0008] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0009] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the method further includes:

[0010] Determine index information of each data block corresponding to the original data;

[0011] An index file is generated according to the index information, and the index file is stored in a disk.

[0012] As an optional implementation manner, determining index information of each data block corresponding to the original data includes:

[0013] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0014] As an optional implementation, when it is determined that at most M data blocks are lost, the method further includes:

[0015] Retrieve at least N data blocks associated with the lost data block;

[0016] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0017] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0018] As an optional implementation manner, retrieving at least N data blocks associated with the lost data block includes:

[0019] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0020] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0021] As an optional implementation, after storing the maximum M restored data blocks in the available disks of the single-node server, the method further includes:

[0022] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0023] As an optional implementation, the method further includes:

[0024] Displays a management interface containing at least one single-node server;

[0025] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0026] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the acquiring the storage parameter of at least one single-node server includes:

[0027] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0028] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0029] As an optional implementation,

[0030] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0031] As an optional implementation,

[0032] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0033] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0034] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and determining, according to the storage parameter of the single-node server, the N first data blocks and the M second data blocks obtained after processing the original data to be stored, includes:

[0035] Slice the original data according to the first parameter to obtain N first data blocks;

[0036] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0037] As an optional implementation manner, encoding the N first data blocks according to the second parameter to obtain M second data blocks includes:

[0038] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0039] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0040] As an optional implementation,

[0041] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0042] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0043] As an optional implementation manner, storing the data blocks corresponding to the original data in different disks of the single-node server respectively includes:

[0044] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0045] In a second aspect, an embodiment of the present disclosure provides a single-node server, including a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and perform the following steps:

[0046] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0047] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0048] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0049] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the processor is further configured to execute:

[0050] Determine index information of each data block corresponding to the original data;

[0051] An index file is generated according to the index information, and the index file is stored in a disk.

[0052] As an optional implementation, the processor is specifically configured to execute:

[0053] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0054] As an optional implementation, when it is determined that at most M data blocks are lost, the processor is further configured to execute:

[0055] Retrieve at least N data blocks associated with the lost data block;

[0056] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0057] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0058] As an optional implementation, the processor is specifically configured to execute:

[0059] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0060] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0061] As an optional implementation, after storing the maximum M restored data blocks in the available disk of the single-node server, the processor is further configured to execute:

[0062] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0063] As an optional implementation, the processor is further configured to execute:

[0064] Displays a management interface containing at least one single-node server;

[0065] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0066] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor is specifically configured to execute:

[0067] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0068] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0069] As an optional implementation,

[0070] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0071] As an optional implementation, the processor is specifically configured to execute:

[0072] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0073] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0074] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor is specifically configured to execute:

[0075] Slice the original data according to the first parameter to obtain N first data blocks;

[0076] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0077] As an optional implementation, the processor is specifically configured to execute:

[0078] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0079] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0080] As an optional implementation,

[0081] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0082] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0083] As an optional implementation, the processor is specifically configured to execute:

[0084] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0085] In a third aspect, an embodiment of the present disclosure further provides a data storage device, the device comprising a processor and a memory, the memory being used to store a program executable by the processor, the processor being used to read the program in the memory and perform the following steps:

[0086] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0087] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0088] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0089] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the processor is further configured to execute:

[0090] Determine index information of each data block corresponding to the original data;

[0091] An index file is generated according to the index information, and the index file is stored in a disk.

[0092] As an optional implementation, the processor is specifically configured to execute:

[0093] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0094] As an optional implementation, when it is determined that at most M data blocks are lost, the processor is further configured to execute:

[0095] Retrieve at least N data blocks associated with the lost data block;

[0096] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0097] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0098] As an optional implementation, the processor is specifically configured to execute:

[0099] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0100] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0101] As an optional implementation, after storing the maximum M restored data blocks in the available disk of the single-node server, the processor is further configured to execute:

[0102] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0103] As an optional implementation, the processor is further configured to execute:

[0104] Displays a management interface containing at least one single-node server;

[0105] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0106] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor is specifically configured to execute:

[0107] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0108] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0109] As an optional implementation,

[0110] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0111] As an optional implementation, the processor is specifically configured to execute:

[0112] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0113] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0114] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor is specifically configured to execute:

[0115] Slice the original data according to the first parameter to obtain N first data blocks;

[0116] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0117] As an optional implementation, the processor is specifically configured to execute:

[0118] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0119] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0120] As an optional implementation,

[0121] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0122] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0123] As an optional implementation, the processor is specifically configured to execute:

[0124] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0125] In a fourth aspect, an embodiment of the present disclosure further provides a data storage device, including:

[0126] An acquisition parameter module, used to acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0127] a slice encoding module, configured to determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0128] The data storage module is used to store the data blocks corresponding to the original data in different disks of the single-node server respectively, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0129] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the data storage module is further used to:

[0130] Determine index information of each data block corresponding to the original data;

[0131] An index file is generated according to the index information, and the index file is stored in a disk.

[0132] As an optional implementation, the data storage module is specifically used for:

[0133] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0134] As an optional implementation, when it is determined that at most M data blocks are lost, the data storage module is further used to:

[0135] Retrieve at least N data blocks associated with the lost data block;

[0136] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0137] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0138] As an optional implementation, the data storage module is specifically used for:

[0139] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0140] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0141] As an optional implementation manner, after storing the maximum M restored data blocks in the available disk of the single-node server, the data storage module is further configured to:

[0142] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0143] As an optional implementation, the interactive unit is further included to:

[0144] Displays a management interface containing at least one single-node server;

[0145] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0146] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the interaction unit is specifically configured to:

[0147] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0148] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0149] As an optional implementation,

[0150] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0151] As an optional implementation manner, the interaction unit is specifically used for:

[0152] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0153] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0154] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the slice encoding module is specifically used for:

[0155] Slice the original data according to the first parameter to obtain N first data blocks;

[0156] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0157] As an optional implementation manner, the slice encoding module is specifically used for:

[0158] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0159] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0160] As an optional implementation,

[0161] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0162] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0163] As an optional implementation, the data storage module is specifically used for:

[0164] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0165] In a fifth aspect, an embodiment of the present disclosure further provides a computer storage medium on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0166] These and other aspects of the present disclosure will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0167] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0168] Figure 1 A specific implementation flow chart of a data storage method provided by an embodiment of the present disclosure;

[0169] Figure 2 A schematic diagram of a hard disk with a common RAID level provided in an embodiment of the present disclosure;

[0170] Figure 3 A schematic diagram of matrix operation of RS coding provided in an embodiment of the present disclosure;

[0171] Figure 4 A schematic diagram of a first-stage data recovery process provided by an embodiment of the present disclosure;

[0172] Figure 5 A schematic diagram of a second-stage data recovery process provided by an embodiment of the present disclosure;

[0173] Figure 6 A schematic diagram of a third-stage data recovery process provided by an embodiment of the present disclosure;

[0174] Figure 7 A schematic diagram of a fourth-stage data recovery process provided by an embodiment of the present disclosure;

[0175] Figure 8 A schematic diagram of a fifth stage data recovery process provided by an embodiment of the present disclosure;

[0176] Fig. 9 An implementation flow chart of an audio and video data storage solution provided in an embodiment of the present disclosure;

[0177] Fig.10 A flowchart of an implementation of an audio and video data recovery solution provided in an embodiment of the present disclosure;

[0178] Fig.11 A management interface provided for an embodiment of the present disclosure;

[0179] Fig.12 A configuration interface for a single-node server provided in an embodiment of the present disclosure;

[0180] Fig.13 A storage parameter configuration completion interface provided in an embodiment of the present disclosure;

[0181] Fig.14 A status monitoring interface provided by an embodiment of the present disclosure;

[0182] Fig.15 A data recovery solution for a single-node server provided in an embodiment of the present disclosure;

[0183] Fig.16 A schematic diagram of a data block storage provided by an embodiment of the present disclosure;

[0184] Fig.17 A schematic diagram of a single-node server provided in an embodiment of the present disclosure;

[0185] Fig.18 A schematic diagram of a data storage device provided by an embodiment of the present disclosure;

[0186] Fig.19 A schematic diagram of a data storage device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0187] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0188] The term "disaster recovery" in the embodiments of the present disclosure generally refers to disaster preparedness, which refers to the use of scientific technical means and methods to establish a systematic data emergency response method in advance to cope with the occurrence of disasters. Its content includes data backup and system backup, business continuity planning, personnel structure, communication guarantee, crisis public relations, disaster recovery planning, disaster recovery plan, business recovery plan, emergency response, third-party cooperation agencies and supply chain crisis management, etc.

[0189] The term "RAID (Redundant Array of Independent Disks)" in the embodiments of the present disclosure is an old disk redundancy backup technology, specifically referring to combining N hard disks into a virtual single large-capacity hard disk through a RAID Controller. Among them, RAID includes RAID0-9, RAID10, etc. according to different implementation technologies. Currently, RAID 5 and RAID 6 are widely used in the field of audio and video storage.

[0190] The term "erasure coding (EC)" in the embodiments of the present disclosure refers to a coding fault-tolerant technology that can add m copies of data based on n copies of original data, and can restore the original data through any n copies of data among n+m copies. That is, if any data less than or equal to m copies fails, it can still be restored through the remaining data.

[0191] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0192] The application scenarios described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. It is known to those skilled in the art that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems. In the description of the present disclosure, unless otherwise specified, the meaning of "multiple" is two or more.

[0193] Embodiment 1. In the face of various possible disasters, enterprises need to conveniently and flexibly synchronize data residing in different databases based on heterogeneous environments. Therefore, it is necessary to build a local and remote disaster recovery system that can resist or resolve various situations. In the application environment of audio and video data storage of traditional single-node servers, it is usually necessary to rely on professional RAID hardware resources to provide multi-level disaster recovery capabilities, such as using RAID5 or RAID6, but the traditional hard RAID solution needs to sacrifice hardware resources to provide highly available disaster recovery capabilities, which is difficult to meet the needs of reducing hardware resource costs and improving disaster recovery capabilities at the same time. In the current single-node audio and video data storage solution, RAID is used to store disaster recovery data. When a disaster occurs, if the stored data is lost or damaged, the original data can be restored through the disaster recovery data stored in the RAID, and then the original data can be stored again in the original storage space, where the disaster recovery data is obtained by processing the original data through preset algorithms and / or operations.

[0194] Traditional hardware RAID solutions are effective and reliable in protecting disk data on a single node. At the same time, implementing RAID algorithms through dedicated computer chips to support hardware can improve efficiency and reduce resource usage, but the hardware cost is relatively high. When using only RAID algorithms that support redundancy, the RAID algorithm can recover a damaged single disk, but the recovery time for 8T+ disks may be as long as several days. At the same time, the flexibility of RAID algorithms in the cloud era is poor.

[0195] The storage solution for audio and video data proposed in this embodiment does not rely on the traditional RAID solution. It implements a multi-level disaster recovery storage solution in the form of software and realizes disaster recovery storage by utilizing the improvement of the storage method. It only needs to set different M values ​​to provide different levels of disaster recovery requirements, thus saving hardware resource costs and improving the requirements for disaster recovery capabilities.

[0196] It should be noted that the data in this embodiment includes but is not limited to audio and video data.

[0197] A data storage method provided in this embodiment is applied to a single-node server, and can realize a storage solution for audio and video data. The original data is sliced ​​and encoded using storage parameters to obtain N first data blocks and M second data blocks obtained after the original data is processed. There is an association relationship between any N data blocks obtained after the original data is processed and the remaining M data blocks. When a maximum of M data blocks are lost, the N data blocks corresponding to the remaining original data are used for recovery to recover the lost data blocks. In addition, by storing the data blocks corresponding to the original data in different disks of the single-node server, the storage pressure of a single disk is reduced and the utilization rate of the storage space is improved. Since M in this embodiment can be set, different levels of disaster recovery requirements can also be provided by setting different M. Compared with the traditional RAID solution, it can provide multiple levels of disaster recovery requirements more flexibly and does not rely on hardware resources.

[0198] like Figure 1 As shown, the specific implementation process of a data storage method provided by an embodiment of the present disclosure is as follows:

[0199] Step 100: Obtain storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0200] It should be noted that this embodiment can provide storage requirements for one or more single-node servers, and the storage strategy of each single-node server is determined according to the corresponding storage parameters. This embodiment can provide different disaster recovery level requirements for each single-node server, and achieve storage requirements of different disaster recovery levels by setting storage parameters. Among them, the storage parameters in this embodiment correspond to the single-node server, and the storage parameters of each single-node server can be set by the user on the WEB side, and the set storage parameters are sent to the corresponding single-node server, so that the single-node server can use the storage parameters to achieve the disaster recovery requirements.

[0201] In some embodiments, the original data in this embodiment includes but is not limited to original audio and video stream data.

[0202] In some embodiments, the storage parameters in this embodiment include but are not limited to a first parameter and a second parameter. Optionally, the first parameter and the second parameter are both calculated by the single-node server itself; or, the first parameter is calculated by the single-node server and the second parameter is input by the user; or, the first parameter and the second parameter are both input by the user.

[0203] In some embodiments, the first parameter is determined based on the number of available disks of a single-node server and the cost of adding disks, and the second parameter is determined based on business requirements.

[0204] In some embodiments, the single-node server can calculate the second parameter based on the business requirements corresponding to the data to be stored, and can also recommend the second parameter to the user for selection. The user can choose the second parameter recommended by the single-node server or re-enter a new second parameter. Similarly, the single-node server can also calculate the first parameter based on its own available disk quantity and disk increase cost, and recommend the first parameter to the user for selection. The user can choose the first parameter recommended by the single-node server or re-enter a new first parameter.

[0205] In some embodiments, a single-node server can receive a second parameter input by a user in a management interface displayed on a WEB side, and then calculate a first parameter based on the number of available disks and the disk increase cost of the single-node server. Optionally, the first parameter represents the total disk space and disk available space of the server provided in the project, which can be predetermined. The second parameter represents the actual requirements of the user in the project. For example, the second parameter is determined based on the disaster recovery level desired by the user. After the second parameter is determined, the M value in this embodiment will be obtained. After the M value is determined, the acceptable value range of the first parameter N will be derived according to the calculation formula, and then the additional storage space cost caused by disaster recovery that the user can accept will be finally determined. The value of the first parameter N in this embodiment.

[0206] In some embodiments, the first parameter represents a parameter used when slicing the original data to be stored, and the second parameter represents a parameter used when encoding the original data to be stored. Optionally, the second parameter is less than or equal to the first parameter. Optionally, the first parameter is N, the second parameter is M, N and M are both positive integers, and M is less than or equal to N.

[0207] In some embodiments, the first parameter and the second parameter have the following relationship:

[0208] The second parameter ≤ (the first parameter + 1) / 2; and / or, the ratio of the sum of the first parameter and the second parameter to the number of disks of the single-node server is an integer. In implementation, the first parameter is used to characterize the number of first data blocks after the original data is sliced, and the second parameter is used to characterize the number of second data blocks added after the first data blocks are encoded. Optionally, the sum of the second parameter and the first parameter can be divided by the number of disks of the single-node server (i.e., the total number of available disks). This ensures that each disk in the single-node server is distributed with data blocks, thereby ensuring load balancing of each disk.

[0209] In some embodiments, M and N have the following relationship:

[0210] M≤(N+1) / 2; and / or, the ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0211] Step 101: Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and both N and M are positive integers, and M is less than or equal to N;

[0212] In some embodiments, the storage parameters include a first parameter and a second parameter, and the original data is sliced ​​and encoded by the following steps:

[0213] (1) slicing the original data according to the first parameter to obtain N first data blocks;

[0214] In implementation, the first parameter can be N, and the original data is cut into N parts to obtain N first data blocks, wherein the slicing process specifically refers to slicing the original data in a certain order. Since the video stream of the camera itself is streaming, but general cloud storage is object storage, it is not an endless streaming file, and the streaming video stream must be cut into files one by one. Therefore, it is usually necessary to slice the streaming file and store it in a specific container format after cutting, such as MP4 and TS format. Among them, the size of each data block obtained by slicing can be the same or different, and this embodiment does not make too many restrictions on this. In this embodiment, the order of slicing the original data is determined based on the principle of correction and erasure, and is not too limited here.

[0215] (2) Encoding the N first data blocks according to the second parameter to obtain M second data blocks.

[0216] In implementation, the second parameter may be M, and after encoding N first data blocks, M second data blocks different from the first data blocks are obtained.

[0217] In some embodiments, the original data may be processed by erasure coding, and the N first data blocks may be erasure coded according to the second parameter to obtain M second data blocks. Optionally, the erasure code in this embodiment includes but is not limited to at least one of array erasure code, RS erasure code and LDPC erasure code.

[0218] Step 102: Store the data blocks corresponding to the original data in different disks of the single-node server respectively, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0219] In some embodiments, the data blocks corresponding to the original data can be stored in different disks according to the load balancing principle and the available storage space of different disks in the single-node server. In implementation, if the available storage space of different disks in the single-node server is different, the load balancing principle can be used to allocate different data blocks to different disks for storage. For example, more data blocks can be allocated to disks with more storage space for storage, and fewer data blocks can be allocated to disks with less storage space for storage, thereby ensuring load balancing of each disk of the single-node server. If the available storage space of different disks in the single-node server is the same, the data blocks corresponding to the original data can be evenly allocated to each disk to ensure load balancing of each disk.

[0220] It should be noted that this embodiment can use erasure coding technology to slice and encode the original data and store it on different disks of a single-node server, so that when a disaster occurs, the association between any N data blocks in the erasure code and the remaining M data blocks can be used to recover the lost data blocks. The original data in the erasure code corresponds to N+M data blocks.

[0221] The implementation of erasure coding technology currently used in the field of audio and video data storage relies on more than three server nodes, and has not been applied to data storage scenarios of single-node servers. Erasure Coding (EC) is a coding fault-tolerant technology that can add m copies of data to n copies of original data, and can restore the original data through any n copies of data among the n+m copies. Based on this principle, the N copies of the first data block obtained after slicing the original data can be added with M copies of the second data block obtained by encoding the N copies of the first data block, and then the original data can be restored based on any N copies of the data blocks among the N+M copies. That is, if any data less than or equal to m copies fails, it can still be restored through the remaining data. EC is mainly used in the fields of storage and digital coding. For example, disk array storage (RAID5, RAID 6), cloud storage (RS), etc. RAID is a special case of EC. Such as Figure 2 As shown, this embodiment provides a schematic diagram of a hard disk of a common RAID level. In a traditional RAID, only limited disk failures are supported. RAID5 only supports one disk failure, RAID6 supports two disk failures, and EC supports multiple disk failures.

[0222] In some embodiments, the application of the erasure code technology in this embodiment in a distributed storage system includes but is not limited to any one or more of the following: array erasure code (Array Code, such as RAID5, RAID6, etc.), RS (Reed-Solomon, Reed-Solomon erasure code) and LDPC (Low Density Parity Check Code, low-density parity check erasure code).

[0223] Taking RS code as an example, the erasure code encoding principle in this embodiment is described as follows:

[0224] RS code is a coding algorithm based on a finite field, also known as a Galois Field. GF(2^w) is used in RS code, where 2^w≥n+m.

[0225] RS code encoding means: given n data blocks D1, D2...Dn, and a positive integer m, RS generates m code blocks C1, C2...Cm based on the n data blocks. RS code decoding means: for any n and m, the original data can be decoded by taking any n blocks from the n original data blocks and m code blocks, that is, RS tolerates at most m data blocks or code blocks being lost at the same time.

[0226] RS encoding and decoding involves matrix inversion, and the Gaussian elimination method is used. It requires real number addition, subtraction, multiplication and division operations, which cannot be applied to binary data with a word length of w. To solve this problem, RS uses the four arithmetic operations defined in the Galois group GF(2^w). The GF(2^w) domain has 2^w values, each of which corresponds to a polynomial of degree less than w, so the four arithmetic operations on the domain are converted into operations in the polynomial space. Addition in the GF(2^w) domain is XOR, and multiplication is implemented by table lookup. Two tables of size 2^w-1 need to be maintained: log table gflog, and inverse log table gfilog.

[0227] Among them, the multiplication formula in the GF domain is a×b=gfilog(gflog(a)+fglog(b))%(2^w-1); among them, gfilog, gflog and fglog all represent operation symbols in the GF domain, gflog represents encoding operation, and fglog represents decoding operation.

[0228] (1) RS code encoding principle.

[0229] RS coding uses words as the encoding and decoding unit. Large data blocks are split into words with a word length of w (usually 8 or 16 bits), and then the words are encoded and decoded. The encoding principle of data blocks is the same as that of word encoding. Taking words as an example, variables Di and Ci represent a word.

[0230] The input data is regarded as a vector D = (D1, D2, ..., Dn), and the encoded data is regarded as a vector (D1, D2, ..., Dn, C1, C2, .., Cm).

[0231] like Figure 3 As shown, this embodiment provides a schematic diagram of matrix operation of RS coding, taking n=5, m=3 as an example, that is, 5 original data blocks D are multiplied by a (n+m)×n matrix B to obtain a (n+m)×1 matrix. According to the characteristics of the matrix, it can be known that the first 5 values ​​in the (n+m)×1 matrix are equal to the values ​​of the 5 original data blocks D, and the last 3 are the calculated check blocks C. Figure 3 The matrix B in is the encoding matrix (also called the generating matrix, distribution matrix, DistributionMatrix), and the encoding matrix needs to satisfy the requirement that any n×n submatrix is ​​invertible. The matrix D is the original data block, and C is the check block obtained by encoding. To facilitate data storage, the upper part of the encoding matrix is ​​the unit matrix (n rows and n columns), and the lower part is the matrix of m rows and n columns. The lower matrix can be selected as the Vandermonde matrix or the Cauchy matrix.

[0232] (2) Principle of RS code data recovery.

[0233] Among them, RS can tolerate at most m data blocks being deleted. The data recovery process is as follows:

[0234] like Figure 4 As shown, this is a schematic diagram of the first stage data recovery process provided by this embodiment, wherein, assuming that data blocks D1, D4, and C2 are lost, the rows corresponding to the lost data blocks / coding blocks are deleted from the coding matrix;

[0235] like Figure 5 As shown, the schematic diagram of the second stage data recovery process provided by this embodiment, wherein according to Figure 3 The RS encoding operation equation shown can obtain the matrix B' and the first process equation.

[0236] like Figure 6 As shown, the schematic diagram of the third stage data recovery process provided by this embodiment, wherein, since the matrix B' is reversible, the inverse matrix of B' is denoted as (B'^-1), then B'×(B'^-1)=I (unit matrix). The inverse matrix of B' is multiplied on the left on both sides of the equation in the first process.

[0237] like Figure 7 As shown, this embodiment provides a schematic diagram of the fourth stage data recovery process, wherein: Figure 6 The equation in is simplified to obtain the calculation formula for the original data D;

[0238] like Figure 8 As shown, the schematic diagram of the fifth stage data recovery process provided by this embodiment, wherein according to Figure 7 The calculation formula in is used to restore the original data D.

[0239] This embodiment adopts erasure coding technology to encode the original data after slicing to obtain the encoded data block (i.e., the second data block), and stores the first data block (obtained after slicing the original data) and the second data block together to achieve the purpose of fault tolerance. The basic idea is to obtain M redundant second data blocks (encoded blocks) by performing certain encoding calculations on the first data blocks of N original data. For N+M data blocks, when any of the M data blocks are wrong, the original data can be restored using the remaining N data blocks through the corresponding reconstruction algorithm. The process of generating the second data block is called encoding, and the process of restoring the lost data block is called decoding.

[0240] This embodiment uses the N+M (where M<=(N+1) / 2) mode in a single-node server to store audio and video file data, and implements a RAID-like function in a single node in a software manner, supporting the damage of up to M disks without affecting data integrity. Where N is the number of slices of the original data, and M is the number of redundant data (encoded data blocks). In the N+M mode, the solution allows up to M disk failures, and the data can still be accessed normally.

[0241] In some embodiments, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the present embodiment further performs the following steps:

[0242] Step 1) determining index information of each data block corresponding to the original data;

[0243] In some embodiments, the index information can be determined based on the identification information of each data block, wherein the identification information of each data block indicates the uniqueness of the data block, and the identification information of the data block can be defined according to a set rule, or can be determined based on the information carried by the data block itself, which is not limited in this embodiment. The identification information of each data block needs to carry the disk information where the data block is stored, so as to obtain the data block stored on the disk.

[0244] In some embodiments, the index information of each data block is determined based on the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block. In implementation, the index information of each data block includes but is not limited to at least three fields, the first field is used to represent the first identification information related to the original data, the second field is used to represent the second identification information related to the second data block, and the third field is used to represent the disk information stored in the data block. For example, taking the first data block as an example, the index information of the first data block can be {audio and video 1, non-second data block, disk 1}; taking the second data block as an example, the index information of the second data block can be {audio and video 2, second data block 1, disk 2}.

[0245] Since the first field in the index information of each data block in the present embodiment contains the first identification information related to the original data, when data is lost and data recovery is performed, the remaining data blocks belonging to the same original data can be retrieved according to the first field in the index information, and according to the second identification information related to the second data block contained in the second field, it can be determined whether the current data block is the original data block or the encoded block obtained by encoding, and according to the disk information contained in the third field, the data blocks of the stored original data can be read from the corresponding disk, thereby using the remaining data blocks for data recovery.

[0246] The index information in this embodiment is used to retrieve each data block corresponding to the same original data. By traversing the index information, each data block corresponding to the original data can be retrieved more conveniently, providing faster support for data recovery.

[0247] Step 2) Generate an index file according to the index information, and store the index file in a disk.

[0248] It should be noted that the index file in this embodiment contains index information of each data block corresponding to at least one original data, and the index file in this embodiment is used to store the index information of the audio and video data with storage requirements in the single-node server, and each data block in each audio and video data corresponds to one index information. Different audio and video data are distinguished by the first identification information related to the original data in the index information.

[0249] This embodiment stores the index file in the disk. The index file is equivalent to a directory file. The storage space occupied by the disk is small. When data is lost, the remaining data blocks of the original data associated with the lost data can be retrieved from the index file, so that the lost data can be restored using the remaining data blocks.

[0250] In some embodiments, when it is determined that at most M data blocks are lost, the present embodiment further performs the following steps to recover the data:

[0251] Step a) retrieving at least N data blocks associated with the lost data block;

[0252] In practice, since there is an association relationship between any N data blocks of the original data and the remaining M data blocks in this embodiment, when M data blocks are lost, the original data can be restored through the remaining N data blocks of the original data. When the lost data blocks are less than M, the original data can be restored through the remaining N data blocks or all data blocks.

[0253] It should be noted that, in this embodiment, the M data blocks lost may all be the first data blocks or the second data blocks, or may include the first data blocks and the second data blocks, and this embodiment does not impose too many restrictions on this.

[0254] In some embodiments, this embodiment can retrieve at least N data blocks associated with the lost data block in the following manner:

[0255] According to the index information in the index file stored in the disk, target index information associated with the index information of the lost data block is determined; according to the disk information in the target index information, at least N data blocks associated with the lost data block are retrieved from the corresponding disk.

[0256] During implementation, since the index information contains the first identification information related to the original data, the remaining data blocks that belong to the same original data as the lost data blocks can be determined from the index information, and the disk locations where the remaining data blocks are stored can be determined based on the disk information contained in the index information, and the corresponding data blocks can be read from the disk at the corresponding locations, thereby using the remaining data blocks to restore the original data.

[0257] Step b) using at least N data blocks associated with the lost data block to perform a data recovery operation on the lost data block, to obtain at most M recovered data blocks;

[0258] In practice, if at most M data blocks are lost, the original data can be restored using the remaining data blocks belonging to the same original data, and the lost data blocks can be stored on the disk again.

[0259] In some embodiments, erasure coding technology may be used to perform data recovery operations on lost data blocks to obtain a maximum of M recovered data blocks.

[0260] Step c) storing at most M copies of the recovered data blocks in the available disks of the single-node server, wherein different copies of the recovered data blocks are stored on different disks.

[0261] In some embodiments, each recovered data block may be stored in different disks according to a load balancing principle and available storage space of different disks in the single-node server.

[0262] In some embodiments, after storing at most M restored data blocks in the available disks of the single-node server, the present embodiment is further used to perform the following process:

[0263] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0264] During implementation, after data is lost, the lost data blocks will be stored again on a new disk after data recovery. In order to ensure that the index information in the index file is the latest, the disk information stored in the recovered data blocks needs to replace the disk information of the data blocks when the data is lost, so the index information of the data blocks is updated; similarly, when the disk is damaged, the data stored in the disk is lost, and the lost data blocks will be stored again on a new disk after data recovery. Therefore, the disk information in the index information of the lost data blocks is updated to the new disk information stored after data recovery.

[0265] like Fig. 9As shown, the embodiment of the present disclosure provides an implementation flow chart of an audio and video data storage solution, and the specific implementation steps of storing audio and video data on a single-node server are as follows:

[0266] Step 900: Determine the storage requirements of the audio and video data received from the user;

[0267] Step 901: Obtain storage parameters, where the storage parameters include a first parameter and a second parameter;

[0268] Step 902: Slice the original data according to the first parameter to obtain N first data blocks; perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0269] Step 903: Store the data blocks corresponding to the original data in different disks of the single-node server respectively;

[0270] Step 904: Determine index information of each data block according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information where each data block is stored;

[0271] Step 905: Generate an index file according to the index information, and store the index file in a disk.

[0272] like Fig.10 As shown, the embodiment of the present disclosure provides an implementation flow chart of an audio and video data recovery solution, and the specific implementation steps for recovering audio and video data on a single-node server are as follows:

[0273] Step 1000: When it is determined that at most M data blocks are lost, target index information associated with the index information of the lost data blocks is determined according to the index information in the index file stored in the disk;

[0274] In some embodiments, when disk corruption is determined, lost data blocks are determined.

[0275] Step 1001: retrieve at least N data blocks associated with the lost data block from the corresponding disk according to the disk information in the target index information;

[0276] Step 1002: Perform a data recovery operation on the lost data block using at least N data blocks associated with the lost data block to obtain at most M recovered data blocks;

[0277] Step 1003: store at most M restored data blocks in available disks of a single-node server, wherein different restored data blocks are stored in different disks;

[0278] Step 1004: Update the index information of the restored data blocks in the index file before the data restoration operation according to the disk information stored in each restored data block.

[0279] This embodiment also provides a management interface, through which users can configure storage parameters accordingly, effectively avoiding problems such as file loss caused by misoperation of users in previous configuration files or script configuration methods. In addition, when configuring storage parameters, users can complete the configuration without learning skills such as editing scripts, which allows users without relevant knowledge to easily operate, providing a more intelligent management and configuration solution.

[0280] In some embodiments, this embodiment provides a user with a more intelligent and portable operation interface through a displayed management interface, which is used to configure the storage parameters and other parameters of a single-node server. The specific process is as follows:

[0281] A management interface including at least one single-node server is displayed; in implementation, the relevant content of one or more single-node servers can be displayed on the management interface, such as Fig.11 As shown, this embodiment provides a management interface, including multiple single-node servers, and displays the name, IP address, node status, data status, number of disks, storage parameters (N+M), storage space utilization, and operations (including viewing, configuration, data recovery, deletion, etc.) of each single-node server.

[0282] This embodiment can uniformly manage all single-node servers for audio and video storage, and display the name, IP address, single-node server status (normal / faulty, where any single-node server is considered to have one or more disk failures), number of disks (this number refers to the number of data disks of the single-node server), storage parameters (first parameter + second parameter), storage space utilization, and corresponding operations (query, add, view, modify configuration, delete, etc.).

[0283] Receive an edit operation performed by a user on a configuration parameter of a single-node server through the management interface, and perform at least one of adding, modifying, deleting, and querying the configuration parameter corresponding to the edit operation. In some embodiments, a user can add, modify, delete, and query any single-node server and its related content in the management interface, and can also add, modify, delete, and query the configuration parameters of any single-node server.

[0284] In some embodiments, the configuration parameters of a single-node server include but are not limited to at least one of the single-node server name, IP address, node status, data status, number of disks, storage parameters, total available storage space, and storage space utilization.

[0285] In some embodiments, the present embodiment may obtain storage parameters of at least one single-node server through the following process, wherein the storage parameters include a first parameter and a second parameter, and the process is as follows:

[0286] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0287] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0288] like Fig.12 As shown, the configuration interface of a single-node server provided in this embodiment, when a new audio and video storage device or a new single-node server is added, corresponding configurations need to be performed according to user needs, including the name, IP address, number of disks, storage parameters, total available storage space, etc. of the single-node server. Taking the first parameter as N and the second parameter as M as an example, the user enters M=3 for the single-node server, and the single-node server automatically calculates N=5 based on the number of available disks and the disk increase cost. After the user clicks the "OK" button, it jumps to the completion interface, as shown in FIG. Fig.13 As shown, this embodiment provides a storage parameter configuration completion interface, prompting the user that the storage parameter configuration of the single-node server has been completed, and the storage and recovery of audio and video data can be achieved according to the storage parameters. The single-node server will execute and monitor storage tasks according to the storage parameters.

[0289] Optionally, the user can also click the "Configure" button of any single-node server on the management interface to enter the configuration interface of the single-node server, and perform corresponding operations such as adding, modifying, and deleting the configuration parameters of the single-node server.

[0290] In some embodiments, Fig.14 As shown, this embodiment also provides a status monitoring interface. When the single-node server loses a data block, the data recovery status of the lost data block can be viewed through the status monitoring interface; or when the disk of the single-node server fails and loses a data block, the data recovery status of the lost data block can be viewed through the status monitoring interface. In implementation, the user can view the data recovery status and data recovery progress of each disk in any single-node server through the status monitoring interface, and can also manually control the start and end of data recovery for each disk.

[0291] Optionally, if the node status of a single-node server is displayed as "fault" on the management interface, it means that the disk of the single-node server is faulty and data is lost. The user can click the "View" button corresponding to the single-node server on the management interface to jump to the status monitoring interface of the single-node server.

[0292] Optionally, the status monitoring interface of a single-node server includes basic information of the single-node server to which each disk belongs (such as name, IP address, etc.), disk information (such as disk code), data recovery status, data recovery progress and percentage, and related operations (start recovery / end recovery).

[0293] like Fig.15 As shown, this embodiment provides a data recovery solution for a single-node server, and the specific implementation process of the solution is as follows:

[0294] Step 1500, displaying a management interface including a single-node server, and receiving a first parameter N of the single-node server input by a user through the management interface, which is 5 and a second parameter M of the single-node server is 3;

[0295] Step 1501: The single-node server divides 24 disks into 3 volumes, each containing 8 disks, according to the calculation rule of number of volumes = total number of disks / (N+M);

[0296] Among them, the total number of disks of a single-node server is 24.

[0297] Step 1502: when storing audio and video files, slice the original audio and video data into five first data blocks, and obtain three second data blocks by encoding the five first data blocks;

[0298] That is, the original data of an audio or video file contains 8 data blocks.

[0299] Step 1503: Store the eight data blocks on different disks respectively.

[0300] like Fig.16 As shown, a schematic diagram of a data block storage provided by the present embodiment, in implementation, the disk is divided into volumes according to the configuration rule of N+M, and the number of volumes = the number of disks / (N+M). For example, when N=5, M=3, and the number of disks is 24, the number of volumes = 3, each volume contains 8 disks, and the above 8 data blocks can be stored on the 8 disks of volume 1 respectively. Among them, N1~N5 represent 5 first data blocks (i.e. original data) obtained by slicing the original audio and video files (i.e. original data), and M1~M3 represent 3 second data blocks (i.e. redundant data) obtained by encoding.

[0301] A data storage method provided in this embodiment is applied to a video storage project, and can realize an audio and video data storage solution when there is no RAID card on a single-node server and the mainboard is not equipped with a RAID control chip, and the single-node server in this embodiment allows the number of damaged disks to be greater than or equal to 3, and the original data can still be restored after the allowed disk is damaged. This embodiment provides a multi-level fault-tolerant data storage, reading and recovery solution in the audio and video data storage application environment of a single-node server. This embodiment solves the problem that the traditional hard RAID solution (such as: RAID5 / RAID6) must rely on special RAID card hardware, and the single-node server allows the number of damaged disks to be ≤2, and the disaster recovery capability is weak. This embodiment no longer relies on professional RAID card hardware resources, and implements a multi-level disaster recovery storage solution in the form of software; because traditional RAID5, RAID6 and other technologies rely on the main control chip on the professional RAID card or the mainboard RAID chip to implement the data storage disaster recovery strategy, and this embodiment can replace the RAID card, and only requires the server's standard CPU to execute the relevant code of the driver program to provide an N+M disaster recovery solution between disks, effectively saving hardware costs. This embodiment also implements the multi-level disaster recovery requirements of a RAID-like mode, and provides multiple levels of audio and video storage data disaster recovery modes such as N+1, N+0, N+2, N+3, and N+M under a single-node server application mode. When the number of damaged disks does not exceed M, this embodiment can ensure that the stored data is not lost, and provides a higher level of data protection than RAID5 and RAID6.

[0302] Example 2: Based on the same inventive concept, the embodiment of the present disclosure also provides a single-node server. Since the server is the server in the method in the embodiment of the present disclosure, and the principle of solving the problem by the server is similar to that of the method, the implementation of the server can refer to the implementation of the method, and the repeated parts will not be repeated.

[0303] like Fig.17 As shown, the server includes a processor 1700 and a memory 1701, wherein the memory 1701 is used to store a program executable by the processor 1700, and the processor 1700 is used to read the program in the memory 1701 and perform the following steps:

[0304] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0305] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0306] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0307] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the processor 1700 is further configured to execute:

[0308] Determine index information of each data block corresponding to the original data;

[0309] An index file is generated according to the index information, and the index file is stored in a disk.

[0310] As an optional implementation manner, the processor 1700 is specifically configured to execute:

[0311] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0312] As an optional implementation, when it is determined that at most M data blocks are lost, the processor 1700 is further configured to execute:

[0313] Retrieve at least N data blocks associated with the lost data block;

[0314] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0315] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0316] As an optional implementation manner, the processor 1700 is specifically configured to execute:

[0317] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0318] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0319] As an optional implementation, after storing the maximum M restored data blocks in the available disks of the single-node server, the processor 1700 is further configured to execute:

[0320] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0321] As an optional implementation manner, the processor 1700 is further configured to execute:

[0322] Displays a management interface containing at least one single-node server;

[0323] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0324] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor 1700 is specifically configured to execute:

[0325] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0326] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0327] As an optional implementation,

[0328] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0329] As an optional implementation manner, the processor 1700 is specifically configured to execute:

[0330] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0331] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0332] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor 1700 is specifically configured to execute:

[0333] Slice the original data according to the first parameter to obtain N first data blocks;

[0334] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0335] As an optional implementation manner, the processor 1700 is specifically configured to execute:

[0336] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0337] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0338] As an optional implementation,

[0339] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0340] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0341] As an optional implementation manner, the processor 1700 is specifically configured to execute:

[0342] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0343] Example 3: Based on the same inventive concept, the embodiment of the present disclosure also provides a data storage device. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0344] like Fig.18 As shown, the device includes a processor 1800 and a memory 1801, wherein the memory 1801 is used to store a program executable by the processor 1800, and the processor 1800 is used to read the program in the memory 1801 and perform the following steps:

[0345] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0346] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0347] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0348] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the processor 1800 is further configured to execute:

[0349] Determine index information of each data block corresponding to the original data;

[0350] An index file is generated according to the index information, and the index file is stored in a disk.

[0351] As an optional implementation manner, the processor 1800 is specifically configured to execute:

[0352] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0353] As an optional implementation, when it is determined that at most M data blocks are lost, the processor 1800 is further configured to execute:

[0354] Retrieve at least N data blocks associated with the lost data block;

[0355] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0356] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0357] As an optional implementation manner, the processor 1800 is specifically configured to execute:

[0358] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0359] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0360] As an optional implementation, after storing the maximum M restored data blocks in the available disks of the single-node server, the processor 1800 is further configured to execute:

[0361] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0362] As an optional implementation manner, the processor 1800 is further configured to execute:

[0363] Displays a management interface containing at least one single-node server;

[0364] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0365] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor 1800 is specifically configured to execute:

[0366] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0367] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0368] As an optional implementation,

[0369] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0370] As an optional implementation manner, the processor 1800 is specifically configured to execute:

[0371] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0372] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0373] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the processor 1800 is specifically configured to execute:

[0374] Slice the original data according to the first parameter to obtain N first data blocks;

[0375] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0376] As an optional implementation manner, the processor 1800 is specifically configured to execute:

[0377] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0378] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0379] As an optional implementation,

[0380] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0381] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0382] As an optional implementation manner, the processor 1800 is specifically configured to execute:

[0383] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0384] Example 4. Based on the same inventive concept, the embodiment of the present disclosure also provides a device for data storage. Since the device is the device in the method in the embodiment of the present disclosure, and the principle of solving the problem by the device is similar to that of the method, based on the same inventive concept, the implementation of the device provided in the embodiment of the present disclosure can refer to the implementation of the method, and the repeated parts will not be repeated.

[0385] like Fig.19 As shown, the device comprises:

[0386] The parameter acquisition module 1900 is used to acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0387] The slice encoding module 1901 is used to determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0388] The data storage module 1902 is used to store the data blocks corresponding to the original data in different disks of the single-node server respectively, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0389] As an optional implementation manner, after storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the data storage module 1902 is further used to:

[0390] Determine index information of each data block corresponding to the original data;

[0391] An index file is generated according to the index information, and the index file is stored in a disk.

[0392] As an optional implementation, the data storage module 1902 is specifically used for:

[0393] The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

[0394] As an optional implementation, when it is determined that at most M data blocks are lost, the data storage module 1902 is further configured to:

[0395] Retrieve at least N data blocks associated with the lost data block;

[0396] Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks;

[0397] A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

[0398] As an optional implementation, the data storage module 1902 is specifically used for:

[0399] Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk;

[0400] At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

[0401] As an optional implementation, after storing the maximum M restored data blocks in the available disk of the single-node server, the data storage module 1902 is further configured to:

[0402] According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

[0403] As an optional implementation, the interactive unit is further included to:

[0404] Displays a management interface containing at least one single-node server;

[0405] An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

[0406] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the interaction unit is specifically configured to:

[0407] receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements;

[0408] The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

[0409] As an optional implementation,

[0410] The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

[0411] As an optional implementation manner, the interaction unit is specifically used for:

[0412] When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or,

[0413] When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

[0414] As an optional implementation manner, the storage parameter includes a first parameter and a second parameter, and the slice encoding module 1901 is specifically used to:

[0415] Slice the original data according to the first parameter to obtain N first data blocks;

[0416] The N first data blocks are encoded according to the second parameter to obtain M second data blocks.

[0417] As an optional implementation manner, the slice encoding module 1901 is specifically used for:

[0418] Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks;

[0419] The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

[0420] As an optional implementation,

[0421] The M and N have the following relationship: M≤(N+1) / 2; and / or,

[0422] The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

[0423] As an optional implementation, the data storage module 1902 is specifically used for:

[0424] According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

[0425] Based on the same inventive concept, an embodiment of the present disclosure further provides a computer storage medium, on which a computer program is stored, wherein the program implements the following steps when executed by a processor:

[0426] Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored;

[0427] Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N;

[0428] The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

[0429] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0430] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that has the functions specified in one or more boxes.

[0431] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0432] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0433] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for storing data, wherein: The method includes: Acquire storage parameters of at least one single-node server, wherein the storage parameters represent parameters used when slicing and encoding the original data to be stored; the storage parameters include a first parameter and a second parameter; Determine, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, wherein any N data blocks among the N first data blocks and the M second data blocks have an association relationship with M data blocks among the N first data blocks and the M second data blocks except the any N data blocks, and N and M are both positive integers, and M is less than or equal to N; The determining, according to the storage parameters of the single-node server, N first data blocks and M second data blocks obtained after processing the original data to be stored, comprises: slicing the original data according to the first parameters to obtain N first data blocks; encoding the N first data blocks according to the second parameters to obtain M second data blocks; The data blocks corresponding to the original data are respectively stored in different disks of the single-node server, wherein the data blocks corresponding to the original data include N first data blocks and M second data blocks.

2. The method according to claim 1, wherein: After storing the data blocks corresponding to the original data in different disks of the single-node server respectively, the method further includes: Determine index information of each data block corresponding to the original data; An index file is generated according to the index information, and the index file is stored in a disk.

3. The method according to claim 2, wherein: The determining the index information of each data block corresponding to the original data includes: The index information of each data block is determined according to the first identification information related to the original data, the second identification information related to the second data block, and the disk information stored in each data block.

4. The method according to any one of claims 1 to 3, wherein: When it is determined that at most M data blocks are lost, the method further includes: Retrieve at least N data blocks associated with the lost data block; Using at least N data blocks associated with the lost data block, a data recovery operation is performed on the lost data block to obtain at most M recovered data blocks; A maximum of M restored data blocks are stored in the available disks of the single-node server, wherein different restored data blocks are stored in different disks.

5. The method according to claim 4, wherein: The retrieving at least N data blocks associated with the lost data block includes: Determine target index information associated with the index information of the lost data block according to the index information in the index file stored in the disk; At least N data blocks associated with the lost data block are retrieved from the corresponding disk according to the disk information in the target index information.

6. The method according to claim 4, wherein: After storing at most M restored data blocks in the available disks of the single-node server, the method further comprises: According to the disk information stored in each recovered data block, the index information of the recovered data block before the data recovery operation in the index file is updated.

7. The method according to any one of claims 1 to 3, 5 to 6, wherein: The method further includes: Displays a management interface containing at least one single-node server; An editing operation performed by a user on a configuration parameter of a single-node server through the management interface is received, and at least one operation of adding, modifying, deleting, and querying is performed on the configuration parameter corresponding to the editing operation.

8. The method according to claim 7, wherein: The storage parameter includes a first parameter and a second parameter, and obtaining the storage parameter of at least one single-node server includes: receiving a second parameter of at least one single-node server input by a user through the management interface, wherein the second parameter is determined according to business requirements; The first parameter is determined according to the number of available disks of the single-node server and the cost of adding disks.

9. The method according to claim 7, wherein: The configuration parameters include at least one of a single-node server name, an IP address, a node status, a data status, a number of disks, storage parameters, a total amount of available storage space, and a storage space utilization rate.

10. The method according to claim 7, wherein: When the single-node server loses a data block, the data recovery status of the lost data block is checked through the status monitoring interface; or, When the disk of the single-node server fails and data blocks are lost, the data recovery status of the lost data blocks is checked through the status monitoring interface.

11. The method according to claim 1, wherein: The encoding process of the N first data blocks according to the second parameter to obtain M second data blocks includes: Perform erasure coding on the N first data blocks according to the second parameter to obtain M second data blocks; The erasure code includes at least one of an array erasure code, a RS erasure code and a LDPC erasure code.

12. The method according to claim 1, wherein: The M and N have the following relationship: M≤(N+1) / 2; and / or, The ratio of the sum of M and N to the number of disks of the single-node server is an integer.

13. The method according to claim 1, wherein: The storing the data blocks corresponding to the original data in different disks of the single-node server respectively includes: According to the load balancing principle and the available storage space of different disks in the single-node server, each data block corresponding to the original data is stored in different disks respectively.

14. A single-node server, wherein: The server comprises a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods of claims 1 to 13.

15. A data storage device, wherein: The device comprises a processor and a memory, wherein the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the steps of any one of the methods of claims 1 to 13.

16. A computer storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images