Storage management method and device of distributed storage system, equipment and storage medium

Abstract

The invention relates to the technical field of electronics, and discloses a storage management method, device and equipment of a distributed storage system.The method comprises the steps that when to-be-stored data sent by a client side is received, the to-be-stored data serves as replica redundancy to be stored in a plurality of nodes; in this way, impact of erasure code conversion in the writing process on IO writing performance is avoided, time consumed for data writing is shortened, when the first preset condition is met, a source node is determined from the multiple nodes storing the copies, the source node calculates erasure codes of all the copies stored by the source node, a migration instruction is sent to the source node, and the data writing efficiency is improved. According to the method, the source node migrates the blocks to the corresponding nodes based on the storage addresses corresponding to the different blocks in the erasure codes of the copies, the copies stored in the nodes can be deleted subsequently, erasure code storage of data is achieved, meanwhile, time consumed for data writing is shortened, and user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic technology, and in particular to a storage management method, device, equipment and storage medium of a distributed storage system. Background Art

[0002] Distributed shared storage refers to the use of distributed algorithms and network communications to connect heterogeneous devices to form a large-scale shared storage system. Erasure Code (EC) is a forward error correction technology (FEC) that is mainly used to avoid packet loss in network transmission. Storage systems use it to improve storage reliability. Compared with multi-copy replication, erasure codes can achieve higher data reliability with less data redundancy, but the encoding method is more complex and requires a lot of calculations. Erasure codes can only tolerate data loss, but not data tampering, which is why they are named.

[0003] In distributed storage systems, in order to save storage costs, write data is often striped and EC redundancy is calculated for the stripe units. Although EC can be exchanged for storage costs, its essence is calculation in exchange. After obtaining the data sent by the client, the EC of the data is calculated and then stored. However, this method calculates EC redundant data blocks by blocking write requests on the IO path. In low-cost storage systems, due to limited computing resources, this operation will seriously affect the performance of write IO, resulting in a longer time for data writing and affecting user experience. Summary of the invention

[0004] In view of this, the present invention provides a storage management method, apparatus, device and storage medium for a distributed storage system to solve the problem in the related art of calculating EC redundant data blocks by blocking write requests on the IO path. In a low-cost storage system, due to limited computing resources, data writing takes a long time, affecting the user experience.

[0005] In a first aspect, the present invention provides a storage management method for a distributed storage system, the method comprising: when data to be stored sent by a client is obtained, the data to be stored is stored as a copy in multiple nodes of the distributed storage system as redundant copies; when a first preset condition is met, a source node is determined from multiple nodes, so that the source node determines the erasure code of the corresponding copy based on each stored copy, and the erasure code includes multiple blocks; storage addresses corresponding to different blocks in each copy are determined; and a migration instruction is sent to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each copy.

[0006] The storage management method of the distributed storage system provided by the present invention stores the data to be stored as replicas in multiple nodes of the distributed storage system as redundant copies; when a first preset condition is met, a source node is determined from the multiple nodes, so that the source node determines the erasure code of the corresponding copy based on each stored copy, and the erasure code includes multiple blocks; the storage addresses corresponding to different blocks in each copy are determined; and a migration instruction is sent to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each copy. The method provided by the present invention, when receiving the data to be stored sent by the client, first stores the data to be stored as a replica in multiple nodes for redundancy, thereby avoiding the impact of the erasure code conversion in the writing process on the IO writing performance, reducing the time consumption of data writing, and when the first preset condition is met, determining the source node from the multiple nodes storing the replicas, the source node calculates the erasure code of each replica stored by it, and sends a migration instruction to the source node, so that the source node migrates each block to the corresponding node based on the storage address corresponding to different blocks in the erasure code of each replica, and the replicas stored in each node can be deleted later, while realizing the erasure code storage of data, reducing the time consumption of data writing and improving the user experience.

[0007] In an optional embodiment, when a preset condition is met, a source node is determined from multiple nodes so that the source node generates a corresponding erasure code based on each stored replica, including: when the replica capacity value written to each node is greater than a preset threshold, each node is set to a closed state; when each node is detected to be in a closed state and a preset replica conversion condition is met, a source node is determined from multiple nodes so that the source node generates a corresponding erasure code based on each stored replica.

[0008] The method provided by this optional implementation mode sets each node to a closed state when the replica capacity value written to each node is greater than a preset threshold value. When the preset replica conversion conditions are met, the source node is controlled to calculate the erasure code of each replica, thereby avoiding the impact of EC conversion consumption on IO write performance.

[0009] In an optional implementation, determining a source node from a plurality of nodes includes: monitoring a load status of each node; and selecting a node with the lowest load among the plurality of nodes as the source node.

[0010] The method provided by this optional implementation manner uses the node with the lowest load among multiple nodes as the source node, which can effectively ensure that the source node can complete the conversion of the erasure code.

[0011] In an optional implementation, the erasure code of each replica is determined by the following steps: dividing each replica according to a preset rule to obtain multiple data blocks of the corresponding replica; calculating the multiple data blocks of each replica to obtain multiple check blocks; superimposing the multiple data blocks and check blocks of each replica to obtain the erasure code of the corresponding replica.

[0012] In an optional embodiment, the step of determining storage addresses corresponding to different blocks in each replica includes: determining multiple target nodes in the distributed storage system based on a second preset condition; sending a control instruction to each target node, the control instruction is used for each target node to generate an erasure code view group corresponding to different replicas, the erasure code view including addresses and summary information of storage nodes corresponding to different blocks of the corresponding replica; determining the storage addresses corresponding to different blocks based on the erasure code view.

[0013] In an optional implementation, a replica management group is stored in each node, and the replica management group includes the storage address and summary information of each replica. The method also includes: when it is monitored that different blocks of each replica are migrated to the corresponding node, deleting each replica and the replica management group in each node; when a migration request for a target block is received, determining whether the target block is damaged; if the target block is in an undamaged state, migrating a copy of the target block to the corresponding node.

[0014] The method provided in this optional implementation manner adopts a copy migration method for non-damaged erasure coded blocks when it is necessary to migrate the target blocks, rather than a computational repair method, thereby improving the performance of EC data repair in a low-cost storage system.

[0015] In a second aspect, the present invention provides a storage management device for a distributed storage system, the device comprising: an acquisition module, for storing the data to be stored as a copy of redundant data in multiple nodes of the distributed storage system when the data to be stored sent by the client is acquired; a first determination module, for determining a source node from multiple nodes when a first preset condition is met, so that the source node determines the erasure code of the corresponding copy based on each stored copy, and the erasure code includes multiple blocks; a second determination module, for determining the storage addresses corresponding to different blocks in each copy; and a sending module, for sending a migration instruction to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each copy.

[0016] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the storage management method of a distributed storage system of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the storage management method for a distributed storage system of the first aspect or any corresponding embodiment thereof.

[0018] In a fifth aspect, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the storage management method of a distributed storage system of the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of a storage management method for a distributed storage system according to an embodiment of the present invention;

[0021] Figure 2 is a flow chart of another storage management method of a distributed storage system according to an embodiment of the present invention;

[0022] Figure 3 is a flow chart of a storage management method of another distributed storage system according to an embodiment of the present invention;

[0023] Figure 4 is a structural block diagram of a storage management device of a distributed storage system according to an embodiment of the present invention;

[0024] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] In related technologies, in order to save storage costs of distributed storage systems, write data is often striped and EC redundancy is calculated for stripe units. Although EC can be exchanged for storage costs, its essence is calculation in exchange. After obtaining the data sent by the client, the EC of the data is calculated and then stored. However, this method calculates EC redundant data blocks by blocking write requests on the IO path. In low-cost storage systems, due to limited computing resources, this operation will seriously affect the performance of write IO, resulting in a longer time for data writing, affecting user experience.

[0027] In view of this, a storage management method for a distributed storage system provided in an embodiment of the present application can be applied to a server to implement storage management of a storage system. The method provided by the present invention, when receiving the data to be stored sent by the client, first stores the data to be stored as a copy redundantly in multiple nodes, thereby avoiding the impact of the erasure code conversion in the writing process on the IO writing performance, reducing the time consumption of data writing, and when the first preset condition is reached, the source node is determined from the multiple nodes storing the copies, the source node calculates the erasure code of each copy it stores, and sends a migration instruction to the source node, so that the source node migrates each block to the corresponding node based on the storage address corresponding to the different blocks in the erasure code of each copy, and the copies stored in each node can be deleted later, while realizing the erasure code storage of data, reducing the time consumption of data writing and improving the user experience.

[0028] According to an embodiment of the present invention, an embodiment of a storage management method for a distributed storage system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0029] In this embodiment, a storage management method of a distributed storage system is provided, which can be used for the above-mentioned server. Figure 1 is a flow chart of a storage management method of a distributed storage system according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0030] Step S101, when the data to be stored sent by the client is obtained, the data to be stored is redundantly stored as copies in multiple nodes of the distributed storage system.

[0031] Exemplarily, the client may be any client terminal, and the data to be stored may be data that needs to be redundantly stored. In an embodiment of the present application, the distributed storage system includes a target number of nodes. In order to achieve redundant storage, the data to be stored may be stored as copies in multiple nodes.

[0032] Step S102: When a first preset condition is met, a source node is determined from multiple nodes, so that the source node determines an erasure code of a corresponding replica based on each stored replica, and the erasure code includes multiple blocks.

[0033] Exemplarily, in an embodiment of the present application, the first preset condition may be that the size of the stored replica exceeds the threshold capacity. When the size of the stored replica exceeds the threshold capacity, a source node is determined from multiple nodes. The embodiment of the present application does not limit the method of determining the source node, as long as the determined source node can implement the erasure code calculation of the replica. The erasure code includes multiple blocks, specifically, the multiple blocks may be k+m blocks, which include k original data blocks and m check blocks.

[0034] Step S103, determining storage addresses corresponding to different blocks in each copy.

[0035] For example, in the embodiment of the present application, the storage address of each block is mainly the node address where the block needs to be stored, which can be a node identifier. The embodiment of the present application does not limit the specific content of the storage address, as long as the migration of the block can be achieved. Different blocks of the same replica have different corresponding storage addresses.

[0036] Step S104 : Send a migration instruction to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each replica.

[0037] Exemplarily, the source node migrates the corresponding blocks to the corresponding nodes based on the storage addresses of the blocks, and then deletes the duplicate data in each node, thereby realizing an EC storage method that saves storage costs. In the embodiment of the present application, since storage resources are generally limited in a low-cost storage system, and the formation of EC requires a large amount of computing resources, EC is formed by asynchronous migration, which reduces the impact of EC formation on write performance.

[0038] The storage management method of the distributed storage system provided in the present embodiment, when receiving the data to be stored sent by the client, first stores the data to be stored as a replica in multiple nodes as redundant copies, thereby avoiding the impact of the erasure code conversion in the writing process on the writing IO writing performance, reducing the time consumption of data writing, and when the first preset condition is met, determining the source node from the multiple nodes storing the replicas, the source node calculates the erasure code of each replica stored by it, and sends a migration instruction to the source node, so that the source node migrates each block to the corresponding node based on the storage address corresponding to different blocks in the erasure code of each replica, and the replicas stored in each node can be deleted later, while realizing the erasure code storage of data, reducing the time consumption of data writing and improving the user experience.

[0039] In this embodiment, a storage management method of a distributed storage system is provided, which can be used for the above-mentioned server. Figure 2 is a flow chart of a storage management method of a distributed storage system according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0040] Step S201: When the data to be stored sent by the client is obtained, the data to be stored is redundantly stored in multiple nodes of the distributed storage system as a copy. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0041] Step S202: When a first preset condition is met, a source node is determined from multiple nodes, so that the source node determines an erasure code of a corresponding replica based on each stored replica, and the erasure code includes multiple blocks.

[0042] Specifically, the above step S202 includes:

[0043] Step S2021, when the replica capacity value written to each node is greater than a preset threshold, each node is set to a closed state.

[0044] Exemplarily, the preset threshold is determined based on the load capacity of the node. The embodiment of the present application does not limit the specific size of the preset threshold, and those skilled in the art can determine it based on demand. In the embodiment of the present application, an EC migration conversion control module is set in the server. The EC migration conversion control module is a task management module that can monitor the resource status of the system in real time by means of watch. When it is detected that the replica capacity value written to each node is greater than the preset threshold, each node storing the replica is in a closed state. When the node is in a closed state, new replica data cannot be written.

[0045] Step S2022, when it is detected that each node is in a closed state and meets the preset replica conversion condition, a source node is determined from multiple nodes so that the source node generates corresponding erasure codes based on each stored replica.

[0046] Exemplarily, the preset replica conversion condition can be a preset replica conversion timing, which can be determined by a person skilled in the art according to the requirements. In the embodiment of the present application, after the EC migration conversion control module sends the EC conversion task, it will synchronously obtain the source node and hard disk load during the conversion. If the source node and hard disk load become high, the EC migration conversion will immediately notify the node to suspend the conversion task.

[0047] In some optional implementations, determining a source node from a plurality of nodes includes:

[0048] Step a1, monitoring the load status of each node. Exemplarily, the resource status of the system is monitored in real time by means of watch, so as to realize the monitoring of the load status of each node.

[0049] Step a2, select the node with the lowest load among the multiple nodes as the source node. Exemplarily, in the embodiment of the present application, selecting the node with the lowest load from the multiple nodes storing the replica as the source node can effectively ensure that the source node can complete the conversion of the erasure code.

[0050] In some optional implementations, the erasure code of each replica is determined by the following steps:

[0051] Step b1, divide each replica according to a preset rule to obtain multiple data blocks corresponding to the replica.

[0052] Exemplarily, in an embodiment of the present application, when each replica writes to a node, the minimum write unit is a chunk, and multiple written chunks are placed together in a preset storage location of the node. All chunk data of the replica are split into equal sizes (if the size is insufficient, add 0) to form data blocks according to the number of EC data blocks. Here, multiple chunks may be allocated to a large block of EC.

[0053] Step b2, calculating multiple data blocks of each replica to obtain multiple check blocks.

[0054] Illustratively, in the embodiment of the present application, after a data block is formed, multiple check blocks are calculated.

[0055] Step b3, superimpose multiple data blocks and check blocks of each replica to obtain the erasure code of the corresponding replica.

[0056] For example, in the embodiment of the present application, multiple check blocks + multiple data blocks are stored in a local temporary file. Furthermore, after the data blocks and check blocks are formed, summary information of each block is calculated at the same time, and the summary information can be an MD5 value.

[0057] Step S203, determine the storage addresses corresponding to different blocks in each copy. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0058] Step S204: Send a migration instruction to the source node, so that the source node migrates each block to the corresponding node based on the storage addresses corresponding to the different blocks of each replica. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0059] In this embodiment, a storage management method of a distributed storage system is provided, which can be used for the above-mentioned server. Figure 3is a flow chart of a storage management method of a distributed storage system according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0060] Step S301: When the data to be stored sent by the client is obtained, the data to be stored is redundantly stored in multiple nodes of the distributed storage system as a copy. Figure 1 Step S201 of the illustrated embodiment will not be described in detail here.

[0061] Step S302: When the first preset condition is met, a source node is determined from multiple nodes, so that the source node determines the erasure code of the corresponding replica based on each stored replica, and the erasure code includes multiple blocks. Figure 1 Step S202 of the illustrated embodiment will not be described in detail here.

[0062] Step S303, determining storage addresses corresponding to different blocks in each copy.

[0063] Specifically, the above step S303 includes:

[0064] Step S3031: determine multiple target nodes in the distributed storage system based on a second preset condition.

[0065] Exemplarily, in an embodiment of the present application, after the erasure code of each replica is generated, a node with low load is selected as the target node from the watch cluster node list.

[0066] Step S3032, sending a control instruction to each target node, the control instruction is used for each target node to generate an erasure code view group corresponding to different replicas, the erasure code view includes the address and summary information of the storage nodes corresponding to different blocks of the corresponding replica.

[0067] Exemplarily, the target node is ordered to generate an erasure code view group for each replica. In the embodiment of the present application, a view group is formed between different EC blocks. EC blocks with redundant relationships form a view group. Each view group has a unique ID number, and the storage address of all EC blocks is identified in the view group. The description of the EC view group is as follows with the MD5 value of the block:

[0068] {ec_cluster_id:x

[0069] view:{

[0070] shard_1:{ip:ip_1;crc:md5_1;};

[0071]

[0072] Among them, view describes the location information of a group of EC members with computational redundancy. The location is generally on different hard disks, and the specific location of the hard disk is determined by the failure domain set by the cluster. The crc value describes the data block content summary value corresponding to the data block.

[0073] Step S3033: determining storage addresses corresponding to different blocks based on the erasure code view.

[0074] Exemplarily, the storage address of each block may be determined based on the erasure code view.

[0075] Step S304: Send a migration instruction to the source node, so that the source node migrates each block to the corresponding node based on the storage addresses corresponding to the different blocks of each replica. Figure 1 Step S204 of the illustrated embodiment will not be described in detail here.

[0076] In some optional implementations, each node stores a replica management group, the replica management group includes a storage address and summary information of each replica, and the method further includes:

[0077] Step S305: When it is detected that different blocks of each replica are migrated to the corresponding node, each replica and the replica management group in each node are deleted.

[0078] Exemplarily, after storing each block of the replica to the corresponding node, redundant storage of the replica corresponding data has been achieved. Deleting the replica data of each node and deleting the replica management group can effectively save storage resources. In the embodiment of the present application, when data is written to the node of the storage system, it is redundantly stored in the form of a replica. The view of the replica management group is as follows:

[0079]

[0080] }

[0081] }

[0082] Among them, replicate represents the storage address of a replica of the replica management group and the md5 value of the content stored in the replica. The consistency of Replicate data is the responsibility of the view management module. Status indicates whether the replica has been asynchronously converted to EC through the migration module. When the EC view group corresponding to the replica management group is formed, the corresponding replica management group will be deleted.

[0083] Step S306: When receiving a migration request for a target block, determine whether the target block is damaged.

[0084] Exemplarily, in an embodiment of the present application, the data detection module is responsible for checking the quality of all EC data blocks of the current node. The detection method is content detection. The data detection module generally chooses to execute in a speed-limited manner, and regularly detects whether the data is damaged. The detection result will be persisted in the local storage state of the EC data block and reported to the superior server of the EC data. An EC data migration module is set in the server. The EC data migration module will persist the damaged data block information reported by each node in the queue storage and mark it as a high-priority task. The EC data migration module provides a migration trigger interface. When the cluster needs to migrate certain EC blocks due to operations such as expansion, reduction, load balancing, and manually triggered migration, the server will also persist the received requests in a task list. The priority of a general task is determined by the task type. Based on the migration request of the target block and the detection result, it can be determined whether the target block is damaged.

[0085] Step S307: If the target block is in an undamaged state, the target block copy is migrated to the corresponding node.

[0086] Exemplarily, in the embodiment of the present application, in order to improve the efficiency of EC repair and reduce the consumption of EC data migration, the EC data migration module will preferentially trigger EC data copy repair for non-damaged migration tasks.

[0087] Furthermore, the EC data migration module triggers the migration of tasks according to the task priority. The EC data migration module will also watch the status of the cluster. When the migration is triggered, it will schedule the execution of the migration task according to the task priority and the cluster status of the cluster watch. Here, it can be defined that the data with the highest priority is generally damaged data, which will be executed first. When the migration task is generated, the machine with low load will be selected as the target machine from the watch cluster results, and the task type can only enable calculation repair. The repair task is defined as:

[0088] Task = {ec_cluster_id:x

[0089] Src_ip:ip_1;

[0090] Dst_ip:ip_2

[0091] }

[0092] Among them, Src_ip is the EC source data block to be migrated, which is in an undamaged state, and Dst_ip is the target address of the migration. When the EC data migration module selects Dst_ip, it also selects it from the cluster status result set of watch. After the task is generated, the EC data migration module will send the migration task to the Dst_ip node. The Dst_ip node sends the copy transmission method to the Src_ip node to copy the data. The data will be stored in the tmp directory of the Dst_ip node. When the data block is copied, Dst_ip verifies whether the crc value of the data block is consistent with the crc value of the corresponding data block recorded in the EC view group. If it is consistent, it will be renamed from tmp to the real storage space. If the copy is completed and the data is inconsistent, the EC data migration module will be notified to regenerate the calculation and repair task. Here, in order to improve the copy success rate, when encountering a transmission problem, the copy task should support breakpoint continuation. When the retry fails X times, the EC data migration module will be notified of the task failure. Finally, the EC migration conversion control module and the EC data migration module can be several different threads of the same process, so that the EC migration situation can be better controlled according to the system resource status.

[0093] The method provided by the embodiment of the present invention controls the progress of EC migration tasks through the EC data migration module. When the source EC data blocks to be migrated are generally not damaged in the scenario of expansion, reduction, or relocation, the EC data migration module controls the copy and repair of the source data blocks when they are not damaged, rather than calculating and repairing them, to improve the performance of EC data migration and reduce the impact of EC calculation and multi-channel data transmission on cluster performance. A high priority is taken to repair damaged data to ensure that damaged data can be repaired as soon as possible.

[0094] In this embodiment, a storage management device for a distributed storage system is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0095] This embodiment provides a storage management device for a distributed storage system. Figure 4 Including:

[0096] The acquisition module 401 is used to store the data to be stored as a copy in multiple nodes of the distributed storage system when the data to be stored sent by the client is acquired;

[0097] A first determination module 402 is used to determine a source node from multiple nodes when a first preset condition is met, so that the source node determines an erasure code of a corresponding replica based on each stored replica, where the erasure code includes multiple blocks;

[0098] The second determination module 403 is used to determine the storage addresses corresponding to different blocks in each replica;

[0099] The sending module 404 is used to send a migration instruction to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each replica.

[0100] In some optional implementations, the first determining module 402 includes:

[0101] A control submodule, used to set each node to a closed state when the replica capacity value written to each node is greater than a preset threshold;

[0102] The first determination submodule is used to determine a source node from multiple nodes when it is detected that each node is in a closed state and meets a preset replica conversion condition, so that the source node generates a corresponding erasure code based on each stored replica.

[0103] In some optional implementations, determining a source node from a plurality of nodes includes:

[0104] Monitor the load status of each node;

[0105] The node with the lowest load among multiple nodes is selected as the source node.

[0106] In some optional implementations, the erasure code of each replica is determined by the following steps: dividing each replica according to a preset rule to obtain multiple data blocks of the corresponding replica; calculating the multiple data blocks of each replica to obtain multiple check blocks; superimposing the multiple data blocks and check blocks of each replica to obtain the erasure code of the corresponding replica.

[0107] In some optional implementations, the second determining module 403 includes:

[0108] A second determination submodule, configured to determine a plurality of target nodes in the distributed storage system based on a second preset condition;

[0109] A sending submodule is used to send a control instruction to each target node, and the control instruction is used for each target node to generate an erasure code view group corresponding to different replicas, and the erasure code view includes the address and summary information of the storage node corresponding to different blocks of the corresponding replica;

[0110] The third determination submodule is used to determine storage addresses corresponding to different blocks based on the erasure code view.

[0111] In some optional implementations, each node stores a replica management group, the replica management group includes a storage address and summary information of each replica, and the device further includes:

[0112] The deletion module is used to delete each replica and the replica management group in each node when it is detected that different blocks of each replica have been migrated to the corresponding node;

[0113] A judgment module, used for judging whether the target block is damaged when receiving a migration request of the target block;

[0114] The migration module is used to migrate the target block copy to the corresponding node if the target block is in an undamaged state.

[0115] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0116] The storage management device of the distributed storage system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0117] The embodiment of the present invention also provides a computer device having the above Figure 4 A storage management device for a distributed storage system is shown.

[0118] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0119] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0120] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0121] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0122] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0123] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0124] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0125] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0126] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A storage management method for a distributed storage system, characterized in that: The method comprises: When the data to be stored sent by the client is obtained, the data to be stored is redundantly stored as a copy in multiple nodes of the distributed storage system; When the first preset condition is met, determining a source node from a plurality of nodes, so that the source node determines an erasure code of a corresponding replica based on each stored replica, wherein the erasure code includes a plurality of blocks; Determine the storage addresses corresponding to different blocks in each replica; A migration instruction is sent to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses respectively corresponding to different blocks of each replica.

2. The method according to claim 1, characterized in that When a preset condition is met, a source node is determined from a plurality of nodes so that the source node generates a corresponding erasure code based on each stored replica, including: When the replica capacity value written to each node is greater than the preset threshold, each node is set to a closed state; When it is detected that each node is in a closed state and meets the preset replica conversion conditions, a source node is determined from multiple nodes so that the source node generates corresponding erasure codes based on each stored replica.

3. The method according to claim 2, characterized in that Determining the source node from the multiple nodes includes: Monitor the load status of each node; The node with the lowest load among multiple nodes is selected as the source node.

4. The method according to claim 2, characterized in that: The erasure code of each replica is determined by the following steps: Divide each replica according to a preset rule to obtain multiple data blocks of the corresponding replica; Calculate multiple data blocks of each replica to obtain multiple check blocks; Multiple data blocks and check blocks of each replica are superimposed to obtain the erasure code of the corresponding replica.

5. The method according to claim 4, characterized in that The step of determining storage addresses corresponding to different blocks in each replica includes: Determining a plurality of target nodes in the distributed storage system based on a second preset condition; Sending a control instruction to each target node, wherein the control instruction is used for each target node to generate an erasure code view group corresponding to different replicas, wherein the erasure code view includes addresses and summary information of storage nodes corresponding to different blocks of the corresponding replica; Storage addresses corresponding to different blocks are determined based on the erasure code view.

6. The method according to any one of claims 1 to 5, characterized in that: Each node stores a replica management group, wherein the replica management group includes a storage address and summary information of each replica. The method further includes: When it is detected that different blocks of each replica have been migrated to the corresponding node, delete each replica and the replica management group in each node; When receiving a migration request for a target block, determining whether the target block is damaged; If the target block is in an undamaged state, the target block copy is migrated to the corresponding node.

7. A storage management device for a distributed storage system, characterized in that: The device comprises: An acquisition module, configured to, when acquiring the data to be stored sent by the client, store the data to be stored as a copy of redundant storage in multiple nodes of the distributed storage system; A first determination module, configured to determine a source node from a plurality of nodes when a first preset condition is met, so that the source node determines an erasure code of a corresponding replica based on each stored replica, wherein the erasure code includes a plurality of blocks; The second determination module is used to determine the storage addresses corresponding to different blocks in each replica; The sending module is used to send a migration instruction to the source node, so that the source node migrates each block to a corresponding node based on the storage addresses corresponding to different blocks of each replica.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the storage management method of the distributed storage system according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the storage management method of a distributed storage system according to any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, and the computer instructions are used to make a computer execute the storage management method of the distributed storage system according to any one of claims 1 to 6.

Images