Data reconstruction management method and device, distributed storage system and storage medium

Abstract

The invention discloses a data reconstruction management method and device, a distributed storage system and a storage medium, and belongs to the field of distributed storage. And when the task number in the target task group (the task group is divided by opposite-end physical nodes) of the new data reconstruction transmission task is not the maximum globally, terminating one data reconstruction transmission task from the task group with the maximum task number, and processing the new data reconstruction transmission task. The distribution of the data reconstruction transmission tasks in the physical nodes is balanced from two aspects of limiting the number of tasks processed by a single physical node at the same time and balancing the network load of each physical node, so that the data reconstruction efficiency is improved, and the resource waste of the physical nodes is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of distributed storage, and particularly relates to a data reconstruction management method and device, a distributed storage system and a storage medium. BACKGROUND

[0002] In a distributed storage system based on a placement group (PG) and an object storage daemon (OSD), when a data object needs to be reconstructed, a data reconstruction transmission task is initiated by a PG instance of the data object, and the data reconstruction transmission task needs to rely on network resources of a physical node for data transmission. In related technologies, there is a lack of a mature data reconstruction management method, which causes the distribution of data reconstruction transmission tasks in physical nodes to be unbalanced, for example, a large number of data reconstruction transmission tasks are concentrated in a certain physical node, which reduces the efficiency of data reconstruction and wastes the resources of other physical nodes.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. SUMMARY

[0004] The present application aims to provide a data reconstruction management method, device, distributed storage system and storage medium. In the present application, when the number of data reconstruction transmission tasks currently not processed reaches a first preset threshold, and the number of tasks in a target task group (the task group is divided according to a peer physical node) of a new data reconstruction transmission task is not the largest globally, a data reconstruction transmission task is terminated from the task group with the largest number of tasks, and the new data reconstruction transmission task is processed. From the aspects of limiting the number of simultaneous processing tasks of a single physical node and balancing the network load of each physical node, the distribution of data reconstruction transmission tasks in physical nodes is balanced, which is beneficial to improving the efficiency of data reconstruction and avoiding the waste of resources of physical nodes.

[0005] To solve the above technical problems, the present application provides a data reconstruction management method applied to any object storage device in a distributed storage system, comprising:

[0006] In response to a processing request for a new data reconstruction transmission task, if the number of data reconstruction transmission tasks currently not processed by the object storage device is equal to a first preset threshold, a peer physical node of the new data reconstruction transmission task is taken as an evaluation node;

[0007] Two physical nodes to which two object storage devices receiving the data reconstruction transmission task belong to are the local physical node and the opposite physical node of the data reconstruction transmission task; the grouping criterion of the task group is that data reconstruction transmission tasks with the same opposite physical node and not processed are grouped into one group; the target task group is a task group in which the opposite physical node of the data reconstruction transmission task is the to-be-evaluated node;

[0008] If the number of tasks in the target task group of the new data reconstruction transmission task is not the global maximum, one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed.

[0009] On the other hand, the if the number of tasks in the target task group of the new data reconstruction transmission task is not the global maximum, one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed includes:

[0010] The number of tasks in the task group with the maximum number of tasks in the object storage device is taken as a target number;

[0011] It is judged whether the number of tasks in the target task group of the new data reconstruction transmission task is equal to the target number;

[0012] If not, the number of tasks in the target task group of the new data reconstruction transmission task is added by one to be taken as a to-be-evaluated number;

[0013] If yes, the process ends;

[0014] It is judged whether the to-be-evaluated number is less than the target number;

[0015] If yes, one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed;

[0016] If not, the process ends.

[0017] On the other hand, after it is judged whether the to-be-evaluated number is less than the target number, before one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed, the data reconstruction management method further includes:

[0018] If the to-be-evaluated number is less than the target number, it is judged whether the difference between the target number and the to-be-evaluated number is greater than a second preset threshold value;

[0019] If greater than the second preset threshold, a step of terminating one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task is performed.

[0020] In another aspect, the step of terminating one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task includes:

[0021] determining whether there is a data reconstruction transmission task with a priority lower than that of the new data reconstruction transmission task in the task group with the largest number of tasks in the object storage device;

[0022] If there is, terminating one data reconstruction transmission task with the lowest priority in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task.

[0023] In another aspect, the step of terminating one data reconstruction transmission task with the lowest priority in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task includes:

[0024] determining whether the data reconstruction transmission task with the lowest priority in the task group with the largest number of tasks in the object storage device is multiple;

[0025] If it is multiple, terminating one data reconstruction transmission task with the lowest priority and the latest start time in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task.

[0026] In another aspect, the first preset threshold of each object storage device in the same physical node in the distributed storage system is the same;

[0027] For any physical node in the distributed storage system, the first ratio of the physical node is equal to the second ratio of the physical node;

[0028] The first ratio of the physical node is the ratio of the first preset threshold of the physical node to the sum of the first preset thresholds of all physical nodes in the distributed storage system, and the second ratio of the physical node is the ratio of the network bandwidth of the physical node to the sum of the network bandwidths of all physical nodes in the distributed storage system.

[0029] In another aspect, the grouping standard of the task group is:

[0030] the peer physical node and the data transmission type of the peer physical node are the same, and the data reconstruction transmission tasks that are not processed are grouped into one group;

[0031] The data transmission type of the opposite physical node includes output data of the opposite physical node and received data of the opposite physical node.

[0032] To solve the above technical problems, the application further provides a data reconstruction management device, comprising:

[0033] The first determining module is configured to, in response to a processing request of a new data reconstruction transmission task, if the number of completed data reconstruction transmission tasks currently processed by the object storage device is equal to a first preset threshold, take the opposite physical node of the new data reconstruction transmission task as a to-be-evaluated node.

[0034] The two physical nodes to which the two object storage devices receiving the data reconstruction transmission task belong to each other are a local physical node and an opposite physical node of the data reconstruction transmission task; the grouping standard of the task group is that data reconstruction transmission tasks with the same opposite physical node and not processed are grouped into one group; and the target task group is a task group in which the opposite physical node of the data reconstruction transmission task is a to-be-evaluated node.

[0035] The first action module is configured to, if the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, terminate one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and process the new data reconstruction transmission task.

[0036] To solve the above technical problems, the application further provides a distributed storage system, comprising a plurality of physical nodes.

[0037] Each object storage device in the physical node is configured to implement the steps of the data reconstruction management method as described above when executing the computer program.

[0038] To solve the above technical problems, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the data reconstruction management method as described above.

[0039] Beneficial effects: the present application provides a data reconstruction management method, considering (1) the task quantity threshold (the first preset threshold) of the object storage device can limit the total number of tasks undertaken by the physical node, (2) replacing the data reconstruction transmission task of the "task quantity globally maximum opposite physical node" with a new data reconstruction transmission task can balance the load between physical nodes, therefore, in the present application, when the number of current unprocessed data reconstruction transmission tasks reaches the first preset threshold, and the task quantity in the target task group (the task group is divided according to the opposite physical node) of the new data reconstruction transmission task is not globally maximum, a data reconstruction transmission task is terminated from the task group with the maximum task quantity, and the new data reconstruction transmission task is processed, from the aspects of limiting the number of simultaneous processing tasks of a single physical node and balancing the network load of each physical node, the distribution of data reconstruction transmission tasks in the physical node is balanced, which is beneficial to improving the efficiency of data reconstruction and avoiding the waste of resources of the physical node.

[0040] The present application also provides a data reconstruction management device, a distributed storage system and a storage medium, which have the same beneficial effects as the data reconstruction management method described above. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the related art and the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 A flowchart of a data reconstruction management method provided by the present application;

[0043] Figure 2 A flowchart of another data reconstruction management method provided by the present application;

[0044] Figure 3 A structure diagram of a data reconstruction management device provided by the present application;

[0045] Figure 4 A structure diagram of a distributed storage system provided by the present application;

[0046] Figure 5 A structure diagram of a computer readable storage medium provided by the present application. DETAILED DESCRIPTION

[0047] The core of the present application is to provide a data reconstruction management method, device, distributed storage system and storage medium, in the present application, when the number of current unfinished data reconstruction transmission tasks reaches a first preset threshold, and the number of tasks in the target task group (the task group is divided according to the opposite physical node) of the new data reconstruction transmission task is not the most globally, a data reconstruction transmission task is terminated from the task group with the most number of tasks, and the new data reconstruction transmission task is processed, from the aspects of the number limitation of simultaneous processing tasks of a single physical node and the balance of network load of each physical node, the distribution of data reconstruction transmission tasks in the physical node is balanced, which is beneficial to improve the efficiency of data reconstruction and avoid the waste of resources of the physical node.

[0048] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0049] Please refer to Figure 1 , Figure 1 A flowchart of a data reconstruction management method provided by the present application is shown, the data reconstruction management method is applied to any object storage device in a distributed storage system, and the data reconstruction management method comprises the following steps.

[0050] S101: In response to a processing request of a new data reconstruction transmission task, if the number of current unfinished data reconstruction transmission tasks of the object storage device is equal to a first preset threshold, the opposite physical node of the new data reconstruction transmission task is taken as an evaluation node.

[0051] Among them, the two physical nodes to which the two object storage devices receiving the data reconstruction transmission task belong to each other are the local physical node and the opposite physical node of the data reconstruction transmission task; the grouping standard of the task group is that the opposite physical nodes are the same and the unfinished data reconstruction transmission tasks are grouped into a group; the target task group is a task group in which the opposite physical node of the data reconstruction transmission task is an evaluation node.

[0052] Specifically, considering the technical problems in the background art above, and in combination with the consideration that (1) the task quantity threshold (first preset threshold) of the object storage device can limit the total number of tasks borne by the physical node, and (2) replacing the data reconstruction transmission task of the “task quantity globally maximum peer physical node” with a new data reconstruction transmission task can balance the load among the physical nodes, in the embodiments of the present application, in response to a processing request for a new data reconstruction transmission task, if the number of data reconstruction transmission tasks currently not processed by the object storage device is equal to the first preset threshold, a peer physical node of the new data reconstruction transmission task is taken as an evaluation node, so as to take the evaluation node as a data basis for the execution of subsequent steps.

[0053] S102: If the number of tasks in the target task group of the new data reconstruction transmission task is not globally maximum, terminate one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device, and process the new data reconstruction transmission task.

[0054] Specifically, in the case where the number of tasks in the target task group of the new data reconstruction transmission task is not globally maximum, if one data reconstruction transmission task in the task group with the maximum number of tasks can be terminated and replaced with the new data reconstruction transmission task, the effect of balancing the load among the physical nodes in the distributed storage system will be achieved, therefore in the embodiments of the present application, in the case where the number of tasks in the target task group of the new data reconstruction transmission task is not globally maximum, one data reconstruction transmission task in the task group with the maximum number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed.

[0055] For example, in a distributed storage system using placement groups (PGs) and object storage daemons (OSDs), the first preset threshold of an OSD is 5. The number of data reconstruction transmission tasks currently not processed by the OSD is 5, and at this time a processing request for a new data reconstruction transmission task is received. The peer physical node of the new data reconstruction transmission task is physical node A, and the number of tasks in the target task group (a group of data reconstruction transmission tasks not processed by the peer physical node A) is 3. After statistics, the task group with the maximum number of tasks in the OSD is the task group of the peer physical node B, and the number of tasks is 5 (globally maximum). Since the number of tasks in the target task group (3) is not globally maximum, one data reconstruction transmission task in the task group of the physical node B is terminated, and the new data reconstruction transmission task is processed.

[0056] The present invention provides a data reconstruction management method, which takes into account that (1) the task number threshold (first preset threshold) of the object storage device can limit the total number of tasks undertaken by the physical node, and (2) replacing the data reconstruction transmission task of "the opposite physical node with the largest number of tasks globally" with a new data reconstruction transmission task can balance the load between physical nodes. Therefore, in the present invention, when the number of currently unprocessed data reconstruction transmission tasks reaches the first preset threshold and the number of tasks in the target task group (task groups are divided by opposite physical nodes) of the new data reconstruction transmission task is not the largest globally, a data reconstruction transmission task from the task group with the largest number of tasks is terminated and the new data reconstruction transmission task is processed. From the two aspects of limiting the number of tasks that can be processed simultaneously by a single physical node and balancing the network load of each physical node, the distribution of data reconstruction transmission tasks on the physical nodes is balanced, which is conducive to improving the efficiency of data reconstruction and avoiding waste of resources of physical nodes.

[0057] Based on the above embodiment:

[0058] As an optional embodiment, if the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, terminating a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task includes:

[0059] S201: The number of tasks in the task group with the largest number of tasks in the object storage device is used as the target number;

[0060] S202: Determine whether the number of tasks in the target task group of the new data reconstruction transmission task is equal to the target number;

[0061] S203: If not, the number of tasks in the target task group of the new data reconstruction transmission task is increased by one as the number to be evaluated;

[0062] If equal, then end;

[0063] S204: Determine whether the quantity to be evaluated is less than the target quantity;

[0064] S205: If it is less than, terminate a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, and process a new data reconstruction transmission task;

[0065] If not less than, then end.

[0066] Specifically, to better illustrate the embodiments of the present invention, please refer to Figure 2 , Figure 2 A flowchart of another data reconstruction management method provided by the present invention.

[0067] Specifically, considering that even if: the number of tasks in the target task group of the new data reconstruction transmission task is equal to the target number (the number of tasks in the task group with the largest number of tasks in the object storage device), but if the new data reconstruction transmission task is selected for processing, it will result in the number of tasks in the target task group being equal to the target number, which is not meaningful for physical node load balancing, therefore, in the embodiment of the application, the number of tasks in the target task group of the new data reconstruction transmission task can be added by one as the to-be-evaluated number under the condition that the number of tasks in the target task group of the new data reconstruction transmission task is not equal to the target number, and it is judged whether the to-be-evaluated number is less than the target number, if yes, one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed.

[0068] Wherein, continuing the scenario in the previous embodiment, the number of tasks in the task group (physical node B task group) with the largest number of tasks in the OSD is 5 (target number). The target task group (physical node A task group) of the new data reconstruction transmission task currently has 3 tasks. It is judged that 3 is not equal to 5, and the to-be-evaluated number is calculated as 3+1=4. It is further judged that 4 is less than 5, so one data reconstruction transmission task in the physical node B task group is terminated, and the new data reconstruction transmission task is processed. If the target task group currently has 5 tasks (equal to the target number 5), the new data reconstruction transmission task is not processed.

[0069] As an optional embodiment, after judging whether the to-be-evaluated number is less than the target number, before terminating one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task, the data reconstruction management method further comprises:

[0070] If less than the target number, it is judged whether the difference between the target number and the to-be-evaluated number is greater than a second preset threshold value;

[0071] If greater than the second preset threshold value, step of terminating one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction transmission task is executed.

[0072] Specifically, to avoid frequent termination of tasks due to small number differences, reduce unnecessary task adjustment, and improve task management stability, a second preset threshold value needs to be set, and the termination operation is only performed when the number difference meets the condition.

[0073] For example, the second preset threshold is set as 2. The target number of the task group with the most tasks in a certain OSD is 6, the target task group of the new data reconstruction transmission task to be evaluated is 3, and the difference between the two is 3>2, which meets the condition. Therefore, one data reconstruction transmission task in the task group with the most tasks is terminated, and the new data reconstruction transmission task is processed. If the target number is 5 and the number to be evaluated is 4, the difference is 1≤2, and the termination operation is not performed.

[0074] As an optional embodiment, terminating one data reconstruction transmission task in the task group with the most tasks in the object storage device and processing the new data reconstruction transmission task include:

[0075] determining whether there is a data reconstruction transmission task with a lower priority than the new data reconstruction transmission task in the task group with the most tasks in the object storage device;

[0076] If there is, terminating the data reconstruction transmission task with the lowest priority in the task group with the most tasks in the object storage device and processing the new data reconstruction transmission task.

[0077] Specifically, considering that the importance of different data reconstruction transmission tasks is different, preferentially retaining tasks with high priority can ensure the smooth progress of the key data reconstruction process and improve the effectiveness of data reconstruction. Therefore, a rule for selecting a terminated task based on priority is set.

[0078] For example, the task group (the opposite physical node C) with the most tasks in a certain OSD contains five data reconstruction transmission tasks, and the priorities are high, medium, medium, low, and low, respectively. The priority of the new data reconstruction transmission task is medium, the target task group number is not the globally largest, and the preceding condition is met. Since there is a data reconstruction transmission task with a lower priority (low) in the task group with the most tasks, one data reconstruction transmission task with a low priority in the task group is terminated, and the new data reconstruction transmission task is processed.

[0079] As an optional embodiment, terminating one data reconstruction transmission task in the task group with the most tasks in the object storage device and processing the new data reconstruction transmission task include:

[0080] determining whether the data reconstruction transmission task with the lowest priority in the task group with the most tasks in the object storage device is multiple;

[0081] If it is multiple, terminating the data reconstruction transmission task with the lowest priority and the latest start time in the task group with the most tasks in the object storage device and processing the new data reconstruction transmission task.

[0082] Specifically, considering that when the number of data reconstruction transmission tasks with the lowest priority is multiple, the termination object needs to be further clarified to reduce the waste of resources already invested, the rule of terminating the task with the latest start time is set.

[0083] For example, in an OSD, the number of data reconstruction transmission tasks with the lowest priority in the maximum task group (opposite physical node D) is 3, and the start times are 09:00, 09:10, and 09:20, respectively. The new data reconstruction transmission task meets all the preceding conditions, so the data reconstruction transmission task with the lowest priority whose start time is 09:20 is terminated, and the new data reconstruction transmission task is processed.

[0084] As an optional embodiment, in the distributed storage system, the first preset threshold of each object storage device in the same physical node is the same;

[0085] For any physical node in the distributed storage system, the first ratio of the physical node is equal to the second ratio of the physical node;

[0086] The first ratio of the physical node is the ratio of the first preset threshold of the physical node to the sum of the first preset thresholds of all physical nodes in the distributed storage system, and the second ratio of the physical node is the ratio of the network bandwidth of the physical node to the sum of the network bandwidths of all physical nodes in the distributed storage system.

[0087] Specifically, considering the factor that the network bandwidth of the physical node determines its ability to carry tasks, the task threshold (first preset threshold) of the node needs to be matched with the bandwidth ratio to avoid overloading of nodes with low bandwidth or idling of nodes with high bandwidth, so the rule of associating the threshold with the bandwidth ratio is set.

[0088] Specifically, for example, the distributed storage system includes 3 physical nodes, and the network bandwidths of the physical nodes are 100 Mbps, 200 Mbps, and 300 Mbps, respectively, and the total network bandwidth is 600 Mbps. Assuming that the sum of the first preset thresholds of all physical nodes is 60, the first ratio of the first physical node = 100 / 600 = 1 / 6, and the first preset threshold = 60 x 1 / 6 = 10; the first ratio of the second physical node = 200 / 600 = 1 / 3, and the first preset threshold = 20; the first ratio of the third physical node = 300 / 600 = 1 / 2, and the first preset threshold = 30. The first preset threshold of all object storage devices in the same physical node is this value.

[0089] As an optional embodiment, the grouping standard of the task group is:

[0090] The opposite physical node and the data transmission type of the opposite physical node are the same, and the data reconstruction transmission tasks that are not processed are grouped into a group;

[0091] The data transmission type of the opposite physical node includes output data of the opposite physical node and received data of the opposite physical node.

[0092] Specifically, considering the factor that the data transmission type (output / received) of the opposite physical node affects the direction of network load of the node, further grouping according to the type can refine the balance of sending and receiving load and avoid overload in a single direction, so the grouping standard of the task group is refined.

[0093] For example, the task group is grouped according to the opposite physical node and the transmission type, and the transmission type includes output data of the opposite physical node and received data of the opposite physical node. In a certain OSD, the opposite physical node is E, and the transmission type of the unprocessed completed data reconstruction transmission task is output data of the opposite physical node, and the task group G-E-output is formed. The opposite physical node is E, and the transmission type of the unprocessed completed data reconstruction transmission task is received data of the opposite physical node, and the task group G-E-receive is formed. The opposite physical node of the new data reconstruction transmission task is E, and the transmission type is output data of the opposite physical node, so the target task group is G-E-output, and whether the number of tasks in the group is globally maximum is evaluated.

[0094] In addition, as an optional embodiment, in response to the processing request of the new data reconstruction transmission task, if the number of unprocessed completed data reconstruction transmission tasks of the object storage device is equal to the first preset threshold value, the opposite physical node of the new data reconstruction transmission task is included as the to-be-evaluated node.

[0095] In response to the processing request of the new data reconstruction transmission task, the data transmission type of the opposite physical node of the new data reconstruction transmission task is determined as the target type.

[0096] It is judged whether the number of data reconstruction transmission tasks in the task pool corresponding to the target type of the object storage device is equal to the first preset threshold value.

[0097] If yes, the opposite physical node of the new data reconstruction transmission task is taken as the to-be-evaluated node.

[0098] Specifically, considering the factor that the consumption of network resources of the physical node by different transmission types (output / received) is different, the task pool is divided according to the type, and the number of tasks is monitored respectively, so that the load in each direction can be accurately controlled, and the excessive concentration of tasks of a certain type can be avoided, so the determination method of the to-be-evaluated node is refined.

[0099] The first preset threshold of a certain OSD is 5, for example, and is divided into an output task pool (corresponding to a receiving data of a peer physical node) and a receiving task pool (corresponding to an output data of a peer physical node) according to a transmission type. The peer of a new data reconstruction transmission task is a physical node F, the transmission type is an output data of a peer physical node (i.e., the physical node F outputs data to the OSD), and the corresponding receiving task pool. If the current task quantity of the receiving task pool is 5 (equal to the first preset threshold), the physical node F is taken as an evaluated node; if the current task quantity is 4 (less than the threshold), the new data reconstruction transmission task is directly processed.

[0100] Please refer to Figure 3 , Figure 3 A structure diagram of a data reconstruction management device provided by the application, the data reconstruction management device comprising:

[0101] The first determining module is configured to, in response to a processing request of a new data reconstruction transmission task, take a peer physical node of the new data reconstruction transmission task as an evaluated node if a current number of data reconstruction transmission tasks not processed by the object storage device is equal to a first preset threshold.

[0102] The two physical nodes to which the two object storage devices receiving the data reconstruction transmission task belong are a local physical node and a peer physical node of the data reconstruction transmission task; the grouping standard of the task group is that data reconstruction transmission tasks with the same peer physical node and not processed are grouped into one group; and the target task group is a task group in which the peer physical node of the data reconstruction transmission task is an evaluated node.

[0103] The first action module is configured to, if the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, terminate a data reconstruction transmission task in a task group with the largest number of tasks in the object storage device and process the new data reconstruction transmission task.

[0104] On the basis of the above embodiment:

[0105] As an optional embodiment, the first action module comprises:

[0106] The second determining module is configured to take the number of tasks in the task group with the largest number of tasks in the object storage device as a target number.

[0107] The first judging module is configured to judge whether the number of tasks in the target task group of the new data reconstruction transmission task is equal to the target number, and if not, trigger the third determining module, and if yes, trigger the ending module.

[0108] The third determining module is configured to take the number of tasks in the target task group of the new data reconstruction transmission task plus one as an evaluated number.

[0109] a termination module, configured to terminate;

[0110] a second judgment module, configured to judge whether the to-be-evaluated quantity is less than the target quantity, if yes, trigger the second action module, if no, trigger the termination module;

[0111] the second action module is configured to terminate one data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, and process a new data reconstruction transmission task.

[0112] As an optional embodiment, the second judgment module is further configured to:

[0113] if the to-be-evaluated quantity is less than the target quantity, trigger the third judgment module;

[0114] The data reconstruction management apparatus further comprises:

[0115] the third judgment module is configured to judge whether the difference between the target quantity and the to-be-evaluated quantity is greater than a second preset threshold, if yes, trigger the second action module.

[0116] As an optional embodiment, the second action module comprises:

[0117] the fourth judgment module is configured to judge whether there is a data reconstruction transmission task with a lower priority than the new data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, if yes, trigger the third action module;

[0118] the third action module is configured to terminate one data reconstruction transmission task with the lowest priority in the task group with the largest number of tasks in the object storage device, and process the new data reconstruction transmission task.

[0119] As an optional embodiment, the third action module comprises:

[0120] the fifth judgment module is configured to judge whether there are multiple data reconstruction transmission tasks with the lowest priority in the task group with the largest number of tasks in the object storage device, if yes, trigger the fourth action module;

[0121] the fourth action module is configured to terminate one data reconstruction transmission task with the lowest priority and the latest task start time in the task group with the largest number of tasks in the object storage device, and process the new data reconstruction transmission task.

[0122] For the data reconstruction management apparatus provided by the embodiments of the present application, please refer to the aforementioned embodiments of the data reconstruction management method, and the embodiments of the present application will not be described here again.

[0123] Please refer to Figure 4 ,Figure 4 A structure schematic diagram of a distributed storage system provided by the present application, the distributed storage system comprising a plurality of physical nodes;

[0124] Each object storage device in the physical node is configured to implement the steps of the data reconstruction management method in the foregoing embodiments when executing the computer program.

[0125] For the distributed storage system provided by the embodiments of the present application, please refer to the foregoing embodiments of the data reconstruction management method, and the embodiments of the present application will not be described here again.

[0126] Please refer to Figure 5 , Figure 5 A structure schematic diagram of a computer readable storage medium provided by the present application, the computer readable storage medium 51 storing a computer program 52, the computer program 52 being configured to implement the steps of the data reconstruction management method when executed by a processor.

[0127] For the computer readable storage medium provided by the embodiments of the present application, please refer to the foregoing embodiments of the data reconstruction management method, and the embodiments of the present application will not be described here again.

[0128] The embodiments of the present application further provide a computer program product, comprising computer programs / instructions, the computer programs / instructions being configured to implement the steps of the data reconstruction management method in the foregoing embodiments when executed by a processor.

[0129] For the computer program product provided by the embodiments of the present application, please refer to the foregoing embodiments of the data reconstruction management method, and the embodiments of the present application will not be described here again.

[0130] In this specification, the various embodiments are described in a progressive manner, with each embodiment focusing on the differences from the other embodiments. Similar or identical parts between the various embodiments may be referred to in conjunction with each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and for relevant parts, reference may be made to the method description. It should also be noted that, in this specification, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or device comprising that element.

[0131] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data reconstruction management method, characterized in that: Any object storage device used in a distributed storage system, including: In response to a processing request for a new data reconstruction transmission task, if the number of unprocessed data reconstruction transmission tasks of the object storage device is equal to a first preset threshold, use a peer physical node of the new data reconstruction transmission task as a node to be evaluated; The two physical nodes of the two object storage devices that receive the data reconstruction transmission task are the local physical node and the opposite physical node of each other's data reconstruction transmission task. The task grouping standard is: the data reconstruction transmission tasks with the same opposite physical node and unfinished processing are grouped together; the target task group is the task group whose opposite physical node of the data reconstruction transmission task in the group is the node to be evaluated. If the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed.

2. The data reconstruction management method according to claim 1, characterized in that: If the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, terminating a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, and processing the new data reconstruction transmission task includes: Taking the number of tasks in the task group with the largest number of tasks in the object storage device as the target number; Determining whether the number of tasks in the target task group of the new data reconstruction transmission task is equal to the target number; If they are not equal, the number of tasks in the target task group of the new data reconstruction transmission task is increased by one as the number to be evaluated; If equal, then end; Determining whether the quantity to be evaluated is less than the target quantity; If it is less than, terminating a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, and processing the new data reconstruction transmission task; If not less than, then end.

3. The data reconstruction management method according to claim 2, characterized in that: After determining whether the number to be evaluated is less than the target number, terminating a data reconstruction and transmission task in the task group with the largest number of tasks in the object storage device, and before processing the new data reconstruction and transmission task, the data reconstruction management method further includes: If it is less than the target number, determining whether the difference between the target number and the number to be evaluated is greater than a second preset threshold; If it is greater than a second preset threshold, the steps are executed: terminating a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device, and processing the new data reconstruction transmission task.

4. The data reconstruction management method according to claim 1, characterized in that: Terminating a data reconstruction and transmission task in the task group with the largest number of tasks in the object storage device and processing the new data reconstruction and transmission task includes: Determine whether there is a data reconstruction transmission task with a lower priority than the new data reconstruction transmission task in the task group with the largest number of tasks of the object storage device; If so, the data reconstruction transmission task with the lowest priority among the task group with the largest number of tasks in the object storage device is terminated, and the new data reconstruction transmission task is processed.

5. The data reconstruction management method according to claim 4, characterized in that: The step of terminating a data reconstruction transmission task with the lowest priority among the task group with the largest number of tasks of the object storage device and processing the new data reconstruction transmission task includes: Determine whether, in the task group with the largest number of tasks of the object storage device, there are multiple data reconstruction transmission tasks with the lowest priority; If there are multiple tasks, the data reconstruction and transmission task with the lowest priority and the latest task start time in the task group with the largest number of tasks of the object storage device is terminated, and the new data reconstruction and transmission task is processed.

6. The data reconstruction management method according to claim 1, characterized in that: In the distributed storage system, the first preset threshold of each object storage device located in the same physical node is the same; For any physical node in the distributed storage system, the first ratio of the physical node is equal to the second ratio; The first ratio of the physical node is: the ratio of the first preset threshold of the physical node to the sum of the first preset thresholds of each physical node in the distributed storage system, and the second ratio of the physical node is: the ratio of the network bandwidth of the physical node to the sum of the network bandwidth of each physical node in the distributed storage system.

7. The data reconstruction management method according to any one of claims 1 to 6, characterized in that: The grouping criteria for the task groups are: The data transmission types of the opposite physical nodes and the opposite physical nodes are the same, and the unprocessed data reconstruction transmission tasks are grouped together; The data transmission type of the opposite physical node includes the opposite physical node outputting data and the opposite physical node receiving data.

8. A data reconstruction management device, characterized in that: include: A first determining module is configured to, in response to a processing request for a new data reconstruction transmission task, determine, if the number of unprocessed data reconstruction transmission tasks currently on the object storage device is equal to a first preset threshold, a peer physical node of the new data reconstruction transmission task as a node to be evaluated; The two physical nodes of the two object storage devices that receive the data reconstruction transmission task are the local physical node and the opposite physical node of each other's data reconstruction transmission task. The task grouping standard is: the data reconstruction transmission tasks with the same opposite physical node and unfinished processing are grouped together; the target task group is the task group whose opposite physical node of the data reconstruction transmission task in the group is the node to be evaluated. The first action module terminates a data reconstruction transmission task in the task group with the largest number of tasks in the object storage device if the number of tasks in the target task group of the new data reconstruction transmission task is not the largest globally, and processes the new data reconstruction transmission task.

9. A distributed storage system, characterized in that: Includes multiple physical nodes; Each object storage device in the physical node is used to implement the steps of the data reconstruction management method according to any one of claims 1 to 7 when executing a computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data reconstruction management method according to any one of claims 1 to 7 are implemented.

Images