Method for processing storage task, and related device

Abstract

A method for processing a storage task, and a related device, which relate to the technical field of computers. The method comprises: during the process of performing, in a storage cluster, hard disk storage on data corresponding to a distributed storage task by means of a plurality of storage nodes, identifying a slow node from among the plurality of storage nodes; determining data to be stored that corresponds to the remaining storage tasks on the slow node, and on the basis of the data to be stored, determining stored data to be transferred; on the basis of a redundant memory topology table, determining a target redundant memory area matching the slow node, and storing, in the target redundant memory area, the stored data to be transferred; and in response to completing the data storage of the data to be stored, generating a message indicating that the storage tasks of the slow node have been completed.

Description

Processing method of storage task and related device

[0001] Cross-reference to related applications

[0002] The present disclosure claims priority to the Chinese patent application No. 202410733429.5, filed on June 06, 2024, and entitled “Processing method of storage task and related device”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the field of computer technology, and in particular, to a processing method of storage task, a processing device of storage task, a computer readable storage medium, an electronic device and a computer program product. BACKGROUND

[0004] With the development of computer technology and network technology, the demand for data storage and access is increasing, and distributed storage technology is widely used in fields such as big data, cloud computing, and Internet of Things, which have large-scale data storage needs.

[0005] In related technologies, distributed storage systems face the contradiction between read-write performance requirements and actual hardware costs. Cloud vendors allocate limited memory specifications to users, and there is unallocated memory in the storage cluster. This part of unallocated memory causes resource waste of the distributed storage system.

[0006] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0007] The purpose of the present disclosure is to provide a processing method of storage task, a processing device of storage task, a computer readable storage medium, an electronic device and a computer program product.

[0008] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0009] According to one aspect of the present disclosure, a processing method of storage task is provided, comprising: identifying a slow node in a plurality of storage nodes during a process in which a storage cluster stores data corresponding to a distributed storage task through the plurality of storage nodes; determining remaining storage tasks on the slow node corresponding to to-be-stored data, determining to-be-transferred storage data according to the to-be-stored data; determining a target redundant memory area matched with the slow node according to a redundant memory topology table, and storing the to-be-transferred storage data to the target redundant memory area; and generating a message that the storage task of the slow node is completed in response to completion of data storage of the to-be-stored data.

[0010] In an embodiment of the present disclosure, after determining the target redundant memory area matched with the slow node, the processing method of the storage task further comprises: obtaining a target memory address of the target redundant memory area and a write address of the original hard disk of the to-be-transferred storage data; establishing an address mapping table between the write address of the original hard disk and the target memory address; and after storing the to-be-transferred storage data into the target redundant memory area, the processing method of the storage task further comprises: in response to the slow node satisfying the data migration condition, migrating the to-be-transferred storage data from the target redundant memory area to the slow node based on the address mapping table.

[0011] In an embodiment of the present disclosure, after storing the to-be-transferred storage data into the target redundant memory area and before migrating the to-be-transferred storage data from the target redundant memory area to the slow node, the processing method of the storage task further comprises: in response to a read request for the to-be-transferred storage data, performing a read operation on the to-be-transferred storage data from the target redundant memory area according to the address mapping table.

[0012] In an embodiment of the present disclosure, the distributed storage task is a non-blocking task.

[0013] In an embodiment of the present disclosure, the processing method of the storage task further comprises: identifying whether there is unallocated memory on each storage node in the storage cluster; determining a storage node with unallocated memory as a redundant existing node; and constructing a redundant memory topology table according to the capacity of the unallocated memory on each redundant existing node, the address information of each redundant existing node, and the connection path information between each redundant existing node.

[0014] In an embodiment of the present disclosure, determining the target redundant memory area matched with the slow node according to the redundant memory topology table comprises: obtaining the redundant memory topology table; wherein the redundant memory topology table comprises redundant memory areas in the storage cluster, storage capacities of each redundant memory area, and adjacent node information of the storage node where each redundant memory area is located; determining a candidate redundant memory area adjacent to the slow node based on the redundant memory topology table; and determining the target redundant memory area from the candidate redundant memory area according to the data volume of the to-be-transferred storage data and the storage capacity of the candidate redundant memory area.

[0015] In an embodiment of the present disclosure, determining the to-be-transferred storage data according to the to-be-stored data comprises: obtaining the task processing efficiency of the slow node, dividing the to-be-transferred storage data into the to-be-transferred storage data and the to-be-kept storage data based on the task processing efficiency of the slow node; and completing the data storage of the to-be-stored data comprises: storing the to-be-transferred storage data into the target redundant memory area and storing the to-be-kept storage data into the write address of the original hard disk.

[0016] In one embodiment of the present disclosure, determining the to-be-transferred storage data according to the to-be-stored data comprises: determining the to-be-stored data as the to-be-transferred storage data; and completing the data storage of the to-be-stored data comprises: storing all the to-be-transferred storage data to the target redundant memory area.

[0017] According to another aspect of the present disclosure, there is provided a processing apparatus for a storage task, comprising: a slow node identification module configured to identify a slow node in a plurality of storage nodes in a process in which the plurality of storage nodes stores data corresponding to a distributed storage task on a hard disk; a data determination module configured to determine to-be-stored data corresponding to a remaining storage task on the slow node, and determine to-be-transferred storage data according to the to-be-stored data; a memory writing module configured to determine a target redundant memory area matching the slow node according to a redundant memory topology table, and store the to-be-transferred storage data to the target redundant memory area; and a message generation module configured to generate a message indicating that the storage task of the slow node is completed in response to completion of the data storage of the to-be-stored data.

[0018] According to still another aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the processing method for a storage task as described above.

[0019] According to yet another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the processing method for a storage task as described above via execution of the executable instructions.

[0020] According to still another aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program being executed by a processor to implement the processing method for a storage task as described above.

[0021] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not intended to limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and serve to explain the principles of the present disclosure. It is readily apparent to one of ordinary skill in the art that the accompanying drawings are merely exemplary of embodiments of the present disclosure and are therefore not to be considered limiting of the scope of the present disclosure.

[0023] FIG. 1 shows a schematic diagram of an exemplary system architecture to which the processing method for a storage task according to embodiments of the present disclosure can be applied.

[0024] FIG. 2 shows a flow chart of a processing method of a storage task according to an embodiment of the present disclosure.

[0025] FIG. 3 shows a flow chart of determining a target redundant memory area in the processing method of the storage task according to an embodiment of the present disclosure.

[0026] FIG. 4 shows a schematic diagram of a system architecture capable of implementing the processing method of the storage task according to an embodiment of the present disclosure.

[0027] FIG. 5 shows a block diagram of a processing device of the storage task according to an embodiment of the present disclosure.

[0028] FIG. 6 shows a block diagram of a computer device for processing a storage task according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Example implementations are now described with reference to the drawings. Example implementations can be implemented in any number of hardware, software, or combinations of hardware and software. Example implementations can be implemented in one or more computer-readable media using, for example, computer-readable instructions and / or data structures. Generally, computer-readable media (which excludes the physical media with which the computer-readable instructions and / or data structures are presented) allows information, such as for example, computer-readable instructions and / or data structures, to be read by a computer. Examples computer-readable media include, but are not limited to: magnetic media such as hard disks; optical media such as Optical Disks; magnetic / optical media such as floppy disks; volatile memory such as DRAM; non-volatile memory such as EPROM; and memory devices such as DRAM, ROM, and Flash. Computer-readable media can be presented here as a schematic representation of a functional entity, which does not necessarily have to correspond to a physically or logically independent entity. The functional entity can be implemented in the form of software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0030] In addition, the accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. In the drawings:

[0031] In addition, the terms "first", "second", and the like, do not denote any quantity or order but are used only for the purpose of description and are not to be construed as indicating or implying such relative importance. Thus, a feature defined with "first", "second", etc. can include one or more of the features implicitly or explicitly. In the description of the present disclosure, the meaning of "a plurality" is at least two, for example, two, three, etc., unless otherwise specifically defined.

[0032] FIG. 1 shows a schematic diagram of an exemplary system architecture to which the processing method of the storage task according to an embodiment of the present disclosure can be applied.

[0033] As shown in FIG. 1, the system architecture can include a server 101, a network 102, and a client 103. The network 102 is a medium for providing a communication link between the client 103 and the server 101. The network 102 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0034] In an example embodiment, the client 103 that performs data transmission with the server 101 can include, but is not limited to, electronic devices such as smartphones, desktop computers, tablets, laptops, smart speakers, digital assistants, AR (Augmented Reality) devices, VR (Virtual Reality) devices, smart wearable devices, etc. Optionally, the operating system running on the electronic device can include, but is not limited to, an Android system, an IOS system, a Linux system, a Windows system, etc.

[0035] The server 101 can be a standalone physical server, a server cluster or a distributed system formed by multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms. In some practical applications, the server 101 can also be a server of a network platform, such as a transaction platform, a live streaming platform, a social platform, or a music platform, etc., which are not limited in the embodiments of the present disclosure. The server can be a single server or a cluster formed by multiple servers, and the specific architecture of the server is not limited in the present disclosure.

[0036] In an example embodiment, the server 101 can be a storage cluster provided by a cloud service provider to provide data storage and access services. The client 103 can be a device or program that communicates with the storage cluster and requests storage services. The user can register and log in to the cloud service provider through the client 103, and then rent storage nodes in the storage cluster to obtain resources allocated by the storage cluster, such as storage space, data bandwidth, and functional services for data processing, etc. In some practical applications, the user can send a storage instruction for a batch of data to the storage cluster through the client 103, and the storage cluster can generate a distributed storage task corresponding to the batch of data, and then divide the batch of data based on the distributed storage task, and store the divided data on multiple storage nodes in the storage cluster. Each storage node can store data in its own hard disk or other designated hard disk storage medium.

[0037] In an example embodiment, the server 101 can implement a process for processing a storage task as follows. During the process of storing data corresponding to a distributed storage task by a plurality of storage nodes in a storage cluster, the server 101 identifies a slow node among the plurality of storage nodes. The server 101 determines to-be-stored data corresponding to remaining storage tasks on the slow node, determines to-be-transferred storage data according to the to-be-stored data, determines a target redundant memory area matching the slow node according to a redundant memory topology table, stores the to-be-transferred storage data to the target redundant memory area, and generates a message indicating that the storage task of the slow node is completed in response to completion of data storage of the to-be-stored data.

[0038] In addition, it should be noted that FIG. 1 only shows an application environment of the storage task processing method provided by the present disclosure. The number of servers 101, networks 102, and clients 103 in FIG. 1 is only illustrative, and any number of clients, networks, and servers can be provided according to actual needs.

[0039] To better enable those skilled in the art to understand the technical solutions of the present disclosure, the steps of the storage task processing method in the example embodiments of the present disclosure will be described in more detail below in conjunction with the accompanying drawings and examples.

[0040] FIG. 2 shows a flowchart of the storage task processing method according to an example embodiment of the present disclosure. The method provided by the present embodiment can be executed by the server 101 or the client 103 as shown in FIG. 1, but the present disclosure is not limited thereto.

[0041] In the following example, the server 101 is taken as an example for illustration.

[0042] As shown in FIG. 2, the storage task processing method provided by the present embodiment can include the following steps.

[0043] In step S201, during the process of storing data corresponding to a distributed storage task by a plurality of storage nodes in a storage cluster, a slow node among the plurality of storage nodes is identified.

[0044] In the processing of the distributed storage task by the storage cluster, the data corresponding to the distributed storage task can be evenly divided into a plurality of sub-data according to the number of storage nodes, and then each storage node is responsible for data persistence of one sub-data. The storage node can store the sub-data into its own hard disk, or store the sub-data into other specified hard disk storage media, such as the hard disk or hard disk drive of other devices, to achieve hard disk storage of the data corresponding to the distributed storage task.

[0045] In this step, the slow nodes can be identified in the process of the storage cluster processing the distributed storage task. The identification of the slow nodes in the plurality of storage nodes can include: monitoring the task processing indicators of each storage node; the task processing indicators include at least one of the following: task processing time, data processing amount, resource utilization; and the storage nodes whose task processing indicators do not meet the corresponding indicator conditions are determined as the slow nodes. The slow nodes can be regarded as the performance bottleneck of the distributed processing system when processing the storage task, and are one of the important factors affecting the tail latency of the distributed storage system.

[0046] For example, if the time length of a certain storage node processing a task is significantly higher than that of other nodes, it may indicate that the node has performance problems or is overloaded, and it may be one of the slow nodes. If the data amount processed by a certain node is much lower than that of other nodes, it may indicate that the node has problems or poor performance, and it may be one of the slow nodes. The resource utilization may include CPU utilization, memory utilization, network bandwidth utilization, etc. If the CPU utilization of a certain storage node is continuously higher than the CPU utilization threshold, it may indicate that the node is overloaded, which will affect the task processing performance of the node, so that the node becomes a slow node. If the memory utilization of a certain storage node is higher than the memory utilization threshold, it may also cause the performance of the node to decrease, so that the node becomes a slow node. If the network bandwidth utilization of a certain storage node is higher than the bandwidth utilization threshold, it may cause the task processing speed of the node to slow down, and even cause network delay, so that the node becomes a slow node.

[0047] In some practical applications, messages that each storage node completes the storage of the sub-data it is responsible for can also be received. When more than a threshold number of nodes among the storage nodes participating in the execution of the distributed storage task have completed the storage of the sub-data, the storage node that has not completed the storage of the data can be determined as a slow node. Alternatively, the number of storage nodes that have not completed the storage of the data can be recorded, and when the number of storage nodes that have not completed the storage of the data decreases from the total number of storage nodes to a preset remaining threshold, the storage nodes that have not completed the storage of the data at this time can be determined as slow nodes. For example, if 20 storage nodes participate in the execution of the distributed storage task, when 19 storage nodes have all completed the storage of the data, the remaining one storage node can be determined as a slow node. Alternatively, when 18 storage nodes have all completed the storage of the data, the remaining two storage nodes can be determined as slow nodes. The remaining threshold can be preset, for example, it can be 1, 2, 3, etc. A percentage can also be preset, for example, it can be 1%, 5%, etc., and then the remaining threshold is determined according to the product of the total number of storage nodes in the storage cluster and the percentage.

[0048] In step S203, the remaining storage tasks on the slow nodes are determined to correspond to the to-be-stored data, and the to-be-transferred storage data is determined according to the to-be-stored data.

[0049] The remaining storage task on the slow node can be a storage task of a remaining sub-data that has not been stored in the sub-data responsible for storage by the node when the node is determined to be a slow node. The to-be-stored data corresponding to the remaining storage task can be the data indicated by the remaining storage task for storage, i.e., the remaining sub-data that has not been stored by the slow node. At least part of the to-be-stored data can be determined as the to-be-migrated storage data.

[0050] When a storage node is determined to be a slow node, the amount of the remaining storage task on the slow node, i.e., the amount of to-be-stored data that has not been stored, can be determined first, and then at least part of the to-be-stored data is determined as the to-be-migrated storage data, so as to prepare a data basis for reducing or removing the amount of storage task of the slow node in the subsequent steps.

[0051] In step S205, the target redundant memory area matched with the slow node is determined according to the redundant memory topology table, and the to-be-migrated storage data is stored in the target redundant memory area.

[0052] In this step, one or more redundant memory areas can be selected in the redundant memory topology table, and one of them can be selected as the target redundant memory area based on the node information of the slow node. For example, the redundant memory area with the shortest delay between the slow node can be selected as the target redundant memory area.

[0053] When writing data (i.e., data storage) to the target redundant memory area, the to-be-migrated storage data can be stored in the target redundant memory area based on the RDMA protocol. The RDMA (Remote Direct Memory Access) protocol is a high-performance network communication protocol. As a data transmission technology, RDMA allows direct memory access between two computers in a computer network without involving the CPU or kernel of the operating system. The data transmission based on RDMA does not require the intervention of the central processing unit, and has the characteristics of high efficiency and low delay. Storing the to-be-migrated storage data in the target redundant memory area can be regarded as offloading the storage task corresponding to the to-be-migrated storage data to the target redundant memory area. The speed of storing data in memory is faster than that of storing data in a hard disk, so offloading the storage task to the target redundant memory area can make the sub-data responsible for storage by the slow node be stored in the storage medium faster, thereby optimizing the storage performance of the slow node.

[0054] In step S207, a message that the storage task of the slow node is completed is generated in response to the to-be-stored data completing data storage.

[0055] The completion of the storage of the to-be-stored data can mean that all the to-be-stored data are stored in the storage medium, which can include at least the target redundant memory area and can also include a hard disk on the slow node. Through the execution of step S205, the slow node can complete the branch task (i.e., the storage of the sub-data allocated to the slow node) in advance, and therefore, after the execution of step S205, a message can be generated to indicate that the storage task of the slow node is completed in response to the completion of the storage of the to-be-stored data. The message can notify an administrator or automatically trigger a subsequent operation, such as continuing to allocate a new storage task to the slow node, pushing the storage cluster to process a next distributed storage task, or further adjusting the system configuration to optimize performance.

[0056] In some embodiments, determining the to-be-transferred storage data according to the to-be-stored data can include obtaining the task processing efficiency of the slow node, dividing the to-be-transferred storage data into the to-be-transferred storage data and the to-be-kept storage data based on the task processing efficiency of the slow node, and based on this, the "completion of the storage of the to-be-stored data" in step S207 can include that the to-be-transferred storage data are all stored in the target redundant memory area and the to-be-kept storage data are all stored in the original hard disk write address.

[0057] In some embodiments, the to-be-transferred storage data can be determined according to the average unloading efficiency of the historical data to the redundant memory area, and then the to-be-transferred storage data can be divided into the to-be-transferred storage data and the to-be-kept storage data based on the task processing efficiency and the average unloading efficiency. For example, assuming that the task processing efficiency is 1 and the average unloading efficiency is 5, 5 / 6 of the to-be-transferred storage data can be taken as the to-be-transferred storage data and 1 / 6 of the to-be-transferred storage data can be taken as the to-be-kept storage data. In this way, the write duration of the write to the target redundant memory area and the write duration of the write to the original hard disk write address can be basically consistent, and the storage of the to-be-stored data can be completed faster.

[0058] In some embodiments, the to-be-stored data are determined as the to-be-transferred storage data, and based on this, the "completion of the storage of the to-be-stored data" in step S207 can include that the to-be-transferred storage data are all stored in the target redundant memory area.

[0059] Directly unloading all the to-be-transferred storage data to the target redundant memory area can not require additional computing resources in the computing unloading phase.

[0060] In addition, in some embodiments, the distributed storage task is a non-blocking task.

[0061] The non-blocking task can be a task that does not block the execution of other tasks or processes when performing a storage operation, and the task does not cause the entire system or application to stall because it needs to wait for a certain storage operation to complete, for example, data backup, asynchronous writing of database logs, and the like.

[0062] The determination can be made when the distributed storage cluster receives a distributed storage task. If the distributed storage task is a non-blocking task, it means that the write operation corresponding to the distributed storage task does not block the overall task pipeline, and the stored data does not need to be used immediately. Therefore, after confirming that the distributed storage task is a non-blocking task, the storage task processing method shown in FIG. 2 can be performed. Since the distributed storage task is a non-blocking task, the storage data to be transferred can be considered as not needing to be used immediately. Therefore, by using the storage task processing method shown in FIG. 2 to offload the storage data to be transferred to the target redundant memory area, the task processing efficiency of the slow node and the overall distributed cluster can be optimized without affecting the processing of other tasks, thereby improving the performance of the storage cluster.

[0063] In some practical applications, the total task data amount of the distributed storage task can also be determined when the distributed storage cluster receives the distributed storage task. If the total task data amount is greater than the data amount threshold, it can be considered that the data amount to be stored by each storage node is large, and the slow node generated during the processing of the distributed storage task is more likely to have a large amount of storage data. Therefore, the slow node needs to be helped to offload tasks, and the storage task processing method shown in FIG. 2 can be performed to relieve the storage pressure of the slow node and improve the processing efficiency of the distributed storage task.

[0064] By using the storage task processing method provided by the present disclosure, the slow node in the hard disk storage of the data in the distributed storage process can be identified, and the remaining storage data corresponding to the storage task on the slow node can be determined. Then, a target redundant memory area matching the slow node is found, and at least part of the data in the storage data (i.e., the storage data to be transferred) is changed to a storage target address, i.e., from writing to a hard disk to writing to a target redundant memory area. After confirming that the storage data to be stored on the slow node is completed, a message indicating that the storage task of the slow node is completed can be generated. It can be seen that, on the one hand, the present scheme can optimize the storage performance of the slow node by identifying the slow node, transferring the task of the slow node, and asynchronously completing the task, thereby optimizing the performance of the overall distributed storage cluster. Moreover, a storage completion message can be generated after the storage data to be stored in the slow node is completed, thereby reducing the tail latency of the distributed storage system and promoting the processing of subsequent tasks of the storage cluster, thereby effectively improving the performance and efficiency of the storage cluster. On the other hand, the present scheme can effectively utilize the redundant memory not allocated to the user, thereby effectively alleviating the resource waste caused by the redundant memory.

[0065] In some embodiments, after determining the target redundant memory area matching the slow node, the processing method of the storage task can further include: obtaining a target memory address of the target redundant memory area, and a write address of the original hard disk of the storage data to be transferred; and establishing an address mapping table between the write address of the original hard disk and the target memory address. The write address of the original hard disk of the storage data to be transferred can be a hard disk address of the slow node, or a hard disk or hard disk medium of another storage node to which the storage data is to be transferred.

[0066] In addition, after storing the storage data to be transferred to the target redundant memory area, the processing method of the storage task can further include: in response to the slow node satisfying a data migration condition, migrating the storage data to be transferred from the target redundant memory area to the slow node based on the address mapping table.

[0067] The data migration condition can be, for example, that the load of the slow node decreases to below a certain threshold, that the network condition improves, or that the storage space on the slow node is sufficient, etc. For example, the network resources and computing resources of the current slow node can be evaluated, and according to different network resources and computing resources, a start time of migrating the storage data to be transferred from the target redundant memory area back to the slow node can be determined. In addition, the size of the data packet during the data migration process can also be determined according to the actual network condition and resource condition.

[0068] In some embodiments, after storing the storage data to be transferred to the target redundant memory area, and before migrating the storage data to be transferred from the target redundant memory area to the slow node, the processing method of the storage task can further include: in response to a read request for the storage data to be transferred, performing a read operation on the storage data to be transferred from the target redundant memory area according to the address mapping table.

[0069] The read operation on the storage data to be transferred from the target redundant memory area can also be performed based on the RDMA protocol.

[0070] FIG. 3 shows a flowchart of a processing method of a storage task in an embodiment of the present disclosure. As shown in FIG. 3, in some embodiments, determining a target redundant memory area matching the slow node according to the redundant memory topology table can include the following steps.

[0071] Step S301: Obtain a redundant memory topology table; wherein the redundant memory topology table includes redundant memory areas in a storage cluster, storage capacities of each redundant memory area, and adjacent node information of a storage node where each redundant memory area is located.

[0072] The redundant memory topology table can be maintained after being initially constructed, or can be constructed in an initial stage of determining to perform a non-blocking distributed storage task.

[0073] In some embodiments, the processing method of the storage task further comprises constructing a redundant memory topology table in the following manner: identifying whether there is unallocated memory on each storage node in the storage cluster; determining the storage nodes with unallocated memory as redundant presence nodes; and constructing the redundant memory topology table according to the capacity of the unallocated memory on each redundant presence node, the address information of each redundant presence node, and the connection path information between each redundant presence node.

[0074] The memory allocation records of the storage cluster can be obtained, and it can be determined according to the memory allocation records whether there is unallocated memory on each storage node and, if so, the capacity of the unallocated memory. The unallocated memory in a redundant presence node can be regarded as a redundant memory area in the storage cluster. The memory allocation records can include the part not rented by the user and the part not used for task offloading by other storage tasks. In addition, for the storage nodes with unallocated memory, the storage nodes with unallocated memory whose capacity is greater than a capacity threshold can be screened out as redundant presence nodes, so as to ensure that the redundant memory area is greater than the capacity threshold, and the efficiency of task offloading can be ensured by avoiding the selection of too many target redundant memory areas for task offloading.

[0075] In addition, if no unallocated memory is found, the original storage scheme can be maintained, that is, the slow nodes are continued to be used to complete data storage. The search frequency (such as searching once every 1 second or once every 5 seconds) can be set, and after no unallocated memory is found for the first time, it can be continuously detected whether there is unallocated memory based on the search frequency, and the unallocated memory can be used for task offloading when it is found.

[0076] In some practical applications, after the redundant memory topology table is constructed for the first time, the redundant memory topology table can be updated in real time. For example, when a new user rents the storage cluster, the capacity of the relevant redundant memory area in the redundant memory topology table can be updated based on the user's rental situation; when other distributed storage tasks use the redundant memory for task offloading, the capacity of the relevant redundant memory area in the redundant memory topology table can be updated; and when the connection relationship between the storage nodes in the storage cluster changes, the connection path information between the storage nodes where the redundant memory areas are located in the redundant memory topology table can be updated.

[0077] In step S303, the candidate redundant memory area adjacent to the slow node is determined based on the redundant memory topology table.

[0078] The redundant memory area with a node distance to the slow node less than or equal to a distance threshold can be selected as a candidate redundant memory area. The distance threshold can be 1 or 2, for example, to ensure that the finally selected target redundant memory area has a node distance to the slow node as small as possible, thereby ensuring the communication efficiency, such as the efficiency of task offloading and the efficiency of data migration.

[0079] In step S305, the target redundant memory area is determined from the candidate redundant memory areas according to the data amount of the storage data to be transferred and the storage capacity of the candidate redundant memory areas.

[0080] The candidate redundant memory area with a storage capacity greater than or equal to the data amount of the storage data to be transferred can be selected as the target redundant memory area. Alternatively, a selection quantity threshold (such as 1, 2, or 3) can be set, and one or more candidate redundant memory areas are selected as the target redundant memory area. The number of these candidate redundant memory areas does not exceed the selection quantity threshold, and the sum of the storage capacities of these candidate redundant memory areas is greater than or equal to the data amount of the storage data to be transferred. In this way, the target redundant memory area for task offloading can be as small as possible, thereby ensuring the communication efficiency and the maintenance consumption of the redundant memory topology table.

[0081] In some practical applications, the score of each redundant memory area can be determined based on the address of the slow node, the data amount of the storage data to be transferred, and the device quality of the node where each redundant memory area is located. The selection score of each redundant memory area is obtained, and finally the selection quantity threshold redundant memory areas with the highest selection score are selected as the target redundant memory area. For example, if the selection quantity threshold is 2, the first two redundant memory areas with the highest selection score are selected as the target redundant memory area.

[0082] In the scoring process, the distance score of each redundant memory area can be determined according to the node distance between each redundant memory area and the slow node (for example, the closer the distance, the higher the score), the capacity score of each redundant memory area can be determined according to the storage capacity of each redundant memory area (for example, the greater the capacity, the higher the score), and the device score of each redundant memory area can be determined according to the device quality of the node where each redundant memory area is located (for example, the more excellent the device quality, the higher the score). The weights of the distance score, the capacity score, and the device score are set, respectively, and then the distance score, the capacity score, and the device score are weighted and summed to obtain the selection score of each redundant memory area.

[0083] FIG. 4 shows a system architecture diagram of a processing method for implementing a storage task according to an embodiment of the present disclosure.

[0084] As shown in FIG. 4, the system implementing the processing method of the storage task can include a storage application 401, a collaborative design module 402, a storage controller 403, a planning and scheduling module 404, an asynchronous processing module 405, a storage node 406, and a redundant memory 407. Among them, the number of storage applications 401 can be one or more, and the number of storage nodes 406 can be one or more. In order to provide persistence and real-time performance, the storage application 401 often needs to save a large amount of data. Distributed storage tasks can be generated for the data that the storage application 401 needs to store, and the distributed storage tasks are sent to the storage controller 403 through the network; then the storage controller 403 divides the distributed storage tasks into equal actual read-write tasks and hands them over to each storage node 406, which can also be regarded as the storage controller 403 dividing the data that the storage application 401 needs to save into equal sub-data and handing them over to each storage node 406 for data persistence.

[0085] For the collaborative design module 402, its main function is to support the user's own storage business in the time delay sensitive business demand, and through the self-defined design interface, the redundant buffer area is written in priority during the hardware IO operation. Specifically, for the distributed storage task generated by the storage application 401, the user can select whether to perform performance optimization on the distributed storage system (i.e., the storage cluster) through the collaborative design module 402, and implement the appropriate optimization processing in the collaborative design module 402. For example, the storage application 401 can be a user's own distributed file system, and the storage cluster in the embodiment can be used to store the log files in the user's own distributed file system. Among them, the user can be provided with an opportunity to choose whether to optimize through the collaborative design module 402, that is, the user can decide whether to start the function of "writing the to-be-transferred storage data in the slow node into the redundant memory 407 in priority" provided by the present disclosure in the collaborative design module 402 according to the business characteristics of the user.

[0086] For the planning and scheduling module 404, it can identify whether there is a slow node in the storage cluster, and can record the idle memory, network and other resources in the storage cluster, create a redundant memory topology table related to the redundant memory 407, and plan and use the redundant memory 407. It can also be responsible for controlling and maintaining the storage optimization task. The redundant memory 407 identified by the planning and scheduling module 404 can include multiple redundant memory areas in the storage cluster. Among them, the planning and scheduling module 404 can consider the communication delay of the redundant memory 407 with the controller and the CPU when planning the redundant memory 407, and take the redundant memory area with the shortest delay in the redundant memory 407 as the best memory area (i.e., the target redundant memory area) for task offloading of the slow node.

[0087] The planning and scheduling module 404 can also maintain the topology information of the storage cluster and update it in real time. In addition, the planning and scheduling module 404 can also identify whether the distributed storage task supports asynchronous read and write. If the planning and scheduling module 404 determines that the distributed storage task supports asynchronous write, a transfer task and an address mapping table are created for the detected slow node. The address mapping table is an address mapping table between the target memory address of the target redundant memory area in the redundant memory 407 and the original hard disk write address of the storage data to be transferred. The process of creating the transfer task can include: determining the to-be-stored data corresponding to the remaining storage task on the slow node, determining the storage data to be transferred according to the to-be-stored data; determining the target redundant memory area matched with the slow node; and creating a transfer task according to the storage data to be transferred and the target redundant memory area, which indicates that the storage data to be transferred is stored in the target redundant memory area.

[0088] For the asynchronous processing module 405, the main purpose of the module is to complete the IO operation related to the slow node, and the read and write of the redundant memory 407 can be performed according to the RDMA protocol. The module can process the data IO asynchronous migration in the target redundant memory area and the target storage medium (i.e., the original hard disk to be written) according to the address mapping table created by the planning and scheduling module 404. When there are multiple slow nodes, the asynchronous processing module 405 can also schedule the multiple transfer tasks generated by the planning and scheduling module 404, and asynchronously write data to the target redundant memory area of each slow node, or migrate the data back to the originally designated target storage medium (e.g., the hard disk of the corresponding slow node) when the slow node meets the data migration condition.

[0089] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not for limiting purposes. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of the processes. In addition, it is also easy to understand that the processes can be executed synchronously or asynchronously, for example, in multiple modules.

[0090] FIG. 5 shows a block diagram of a storage task processing apparatus 500 according to an embodiment of the present disclosure. As shown in FIG. 5, the apparatus includes: a slow node identification module 501 configured to identify a slow node in a plurality of storage nodes during a process in which the storage cluster stores data corresponding to a distributed storage task by the plurality of storage nodes on a hard disk; a data determination module 502 configured to determine to-be-stored data corresponding to a remaining storage task on the slow node, and determine storage data to be transferred according to the to-be-stored data; a memory write module 503 configured to determine a target redundant memory area matched with the slow node according to a redundant memory topology table, and store the storage data to be transferred in the target redundant memory area; and a message generation module 504 configured to generate a message indicating that the storage task of the slow node is completed in response to the to-be-stored data completing data storage.

[0091] The processing apparatus for storage tasks provided by the embodiments of the present disclosure can identify a slow node in a distributed storage process in which data is stored on a hard disk, then determine the to-be-stored data corresponding to the remaining storage tasks on the slow node, find a matching target redundant memory area for the slow node, then change the storage target address of at least part of the data (i.e. to-be-transferred storage data) in the to-be-stored data from writing to the hard disk to writing to the target redundant memory area, and after confirming that the to-be-stored data on the slow node completes data storage, a message indicating that the storage task of the slow node is completed can be generated. It can be seen that, on the one hand, the present solution can optimize the storage performance of the slow node through asynchronous optimization by identifying the slow node and transferring and asynchronously completing the tasks of the slow node, thereby achieving performance optimization for the entire distributed storage cluster; and a storage completion message can be generated after the to-be-stored data in the slow node completes data storage, thereby reducing the tail latency of the distributed storage system and promoting the storage cluster to process subsequent tasks, effectively improving the performance and efficiency of the storage cluster. On the other hand, the present solution can effectively utilize the redundant memory not allocated to users, effectively alleviating the resource waste caused by this part of memory.

[0092] In some embodiments, the processing apparatus for storage tasks further includes an address mapping table establishing module 505 configured to, after determining the target redundant memory area matching the slow node, obtain a target memory address of the target redundant memory area and an original hard disk write address of the to-be-transferred storage data, and establish an address mapping table between the original hard disk write address and the target memory address; and a migration module 506 configured to, after storing the to-be-transferred storage data to the target redundant memory area, in response to the slow node satisfying a data migration condition, migrate the to-be-transferred storage data from the target redundant memory area to the slow node based on the address mapping table.

[0093] In some embodiments, the processing apparatus for storage tasks further includes a memory reading module 507 configured to, after storing the to-be-transferred storage data to the target redundant memory area and before migrating the to-be-transferred storage data from the target redundant memory area to the slow node, in response to a read request for the to-be-transferred storage data, perform a read operation on the to-be-transferred storage data from the target redundant memory area according to the address mapping table.

[0094] In some embodiments, the distributed storage task is a non-blocking task.

[0095] In some embodiments, the processing device storing the task further comprises a topology table construction module 508 configured to: identify whether there is unassigned memory on each storage node in the storage cluster; determine the storage nodes with unassigned memory as redundant presence nodes; and construct a redundant memory topology table according to the capacity of the unassigned memory on each redundant presence node, the address information of each redundant presence node, and the connection path information between each redundant presence node.

[0096] In some embodiments, the memory writing module 503 determines the target redundant memory area matching the slow node according to the redundant memory topology table, comprising: obtaining the redundant memory topology table; wherein the redundant memory topology table comprises redundant memory areas in the storage cluster, the storage capacity of each redundant memory area, and the adjacent node information of the storage node where each redundant memory area is located; determining a candidate redundant memory area adjacent to the slow node based on the redundant memory topology table; and determining the target redundant memory area from the candidate redundant memory area according to the data volume of the storage data to be transferred and the storage capacity of the candidate redundant memory area.

[0097] In some embodiments, the data determining module 502 determines the storage data to be transferred according to the data to be stored, comprising: obtaining the task processing efficiency of the slow node, and dividing the storage data to be transferred based on the task processing efficiency of the slow node into the storage data to be transferred and the storage data to be retained; and completing the data storage of the data to be stored, comprising: storing the storage data to be transferred to the target redundant memory area, and storing the storage data to be retained to the original hard disk write address.

[0098] In some embodiments, the data determining module 502 determines the storage data to be transferred according to the data to be stored, comprising: determining the data to be stored as the storage data to be transferred; and completing the data storage of the data to be stored, comprising: storing the storage data to be transferred to the target redundant memory area.

[0099] Other contents of the embodiment of FIG. 5 can refer to the above-mentioned other embodiments.

[0100] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combined with hardware and software, which can be collectively referred to as "circuitry", "module" or "system".

[0101] FIG. 6 shows a structural block diagram of a storage task processing computer device in an embodiment of the present disclosure. It should be noted that the electronic device shown is only an example and should not limit the function and use range of the embodiment of the present application.

[0102] The electronic device 600 according to this embodiment of the present application will be described below with reference to FIG. 6. FIG. 6 shows the electronic device 600 as merely an example, and should not impose any limitation on the functions and the usage range of the embodiments of the present application.

[0103] As shown in FIG. 6, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 can include, but are not limited to, the at least one processing unit 610, the at least one storage unit 620, and a bus 630 connecting different system components, including the storage unit 620 and the processing unit 610.

[0104] The storage unit stores program codes which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present application described in the "Exemplary Method" section of the present specification. For example, the processing unit 610 can execute the method as shown in FIG. 2.

[0105] The storage unit 620 can include a readable medium in the form of a volatile storage unit, such as a random access memory (RAM) 6201 and / or a cache memory 6202, and can further include a read-only memory (ROM) 6203.

[0106] The storage unit 620 can further include program / utility 6204 having a set of the program modules 6205, such as an operating system, one or more application programs, other program modules, and program data, and each of these examples, or some combination thereof, can include implementation of a network environment.

[0107] The bus 630 can be one or more of several types of bus structures, including a storage unit bus or bus controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus architectures.

[0108] The electronic device 600 can also communicate with one or more external devices 700 such as a keyboard or pointing device, a Bluetooth device, or a database, and can communicate with one or more devices enabling user interaction with the electronic device 600 (for example, a display, speakers, a microphone, etc.) and / or one or more devices enabling communication of the electronic device 600 with other computing devices. The communication can be facilitated via an input / output (I / O) interface 650. Still yet, the electronic device 600 can communicate with one or more networks, such as one or more local area networks (LANs), wide area networks (WANs), and / or the Internet, through a network adapter 660. As depicted, the network adapter 660 can communicate with the other components of the electronic device 600 via the bus 630. It should be appreciated that the electronic device 600 can be a part of one or more components of a larger computer system, which can include a plurality of computers interconnected via a network.

[0109] In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer program product, which includes a computer program that, when executed by a processor, implements the processing method of the above-mentioned storage task.

[0110] In the exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above-mentioned method of the present disclosure is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code for causing a terminal device to perform the steps described in the above-mentioned “Exemplary Method” section of the present disclosure according to various exemplary embodiments of the present disclosure when the program product is run on the terminal device.

[0111] The program product for implementing the above-mentioned method according to the embodiments of the present disclosure can take the form of a portable compact disc read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited to this, and in the present document, a readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0112] The program product can take any combination of one or more computer-readable media. The computer-readable media can be a computer-readable storage medium or a computer-readable signal medium. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] The computer-readable signal medium can include a computer-readable storage medium that is propagated as a carrier wave. The computer-readable signal medium can further be any computer-readable medium that is not a storage medium. The computer-readable signal medium can be a computer-readable storage medium that is a propagated signal on a computer-readable storage medium.

[0114] The program code embodied on the computer-readable media can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0115] The program code can be executed by one or more programmable processors, which can be individual or grouped processors, to perform the operations of the aspects. The program code can execute entirely on a user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider.

[0116] It should be noted that, although the above detailed description refers to several modules or units of the device for action execution, such a division is not mandatory. Indeed, according to an embodiment of the present disclosure, features and functionalities of two or more modules or units described above can be embodied in one module or unit. Conversely, features and functionalities of one module or unit described above can be further divided into several modules or units embodied.

[0117] Moreover, although individual steps of the methods in the present disclosure are described in a particular order in the drawings, this is not required or implied as to the order of the steps or that all of the steps shown must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, one step can be broken into multiple steps, etc.

[0118] Those skilled in the art will readily understand that the example embodiments described herein can be implemented by software and / or by hardware coupled with software, as described above. Thus, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product. The software product can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash disk, a mobile hard disk, or the like) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to perform the methods according to the embodiments of the present disclosure.

[0119] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the content of the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure following the general principles thereof and including such modifications as would be apparent to those skilled in the art to one of ordinary skill in the art. The specification and examples are to be regarded as illustrative only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A processing method of storing a task, wherein, The method comprises: In the process that a storage cluster stores data corresponding to a distributed storage task through a plurality of storage nodes, a slow node in the plurality of storage nodes is identified; Remaining storage data corresponding to a remaining storage task on the slow node is determined, and the remaining storage data is determined as to-be-transferred storage data according to the remaining storage data; According to a redundant memory topology table, a target redundant memory area matched with the slow node is determined, and the to-be-transferred storage data is stored in the target redundant memory area; In response to completion of data storage of the to-be-transferred storage data, a message that a storage task of the slow node is completed is generated.

2. The method of claim 1, wherein, After determining the target redundant memory area matched with the slow node, the method further comprises: Obtaining a target memory address of the target redundant memory area and an original hard disk write address of the to-be-transferred storage data; Establishing an address mapping table between the original hard disk write address and the target memory address; and After storing the to-be-transferred storage data in the target redundant memory area, the method further comprises: in response to the slow node meeting a data migration condition, migrating the to-be-transferred storage data from the target redundant memory area to the slow node based on the address mapping table.

3. The method of claim 2, wherein, After storing the to-be-transferred storage data in the target redundant memory area and before migrating the to-be-transferred storage data from the target redundant memory area to the slow node, the method further comprises: In response to a read request for the to-be-transferred storage data, performing a read operation on the to-be-transferred storage data from the target redundant memory area according to the address mapping table.

4. The method according to any one of claims 1 to 3, wherein, The distributed storage task is a non-blocking task.

5. The method of claim 1, wherein, The method further comprises: Identifying whether there is unallocated memory on each storage node in the storage cluster; Determining a storage node with unallocated memory as a redundant existing node; According to the capacity of unallocated memory on each redundant existing node, address information of each redundant existing node, and connection path information between each redundant existing node, a redundant memory topology table is constructed.

6. The method of claim 1 or 5, wherein, Determining the target redundant memory area matched with the slow node according to the redundant memory topology table comprises: Obtaining a redundant memory topology table; wherein the redundant memory topology table comprises redundant memory areas in the storage cluster, storage capacities of each redundant memory area, and adjacent node information of a storage node where each redundant memory area is located; Determining a candidate redundant memory area adjacent to the slow node based on the redundant memory topology table; According to the data volume of the to-be-transferred storage data and the storage capacity of the candidate redundant memory area, determining the target redundant memory area from the candidate redundant memory area.

7. The method of claim 1, wherein, Determining the to-be-transferred storage data according to the to-be-stored data comprises: Obtaining a task processing efficiency of the slow node, and dividing the to-be-transferred storage data into to-be-transferred storage data and to-be-kept storage data based on the task processing efficiency of the slow node; and The to-be-stored data completes data storage, which comprises that the to-be-transferred storage data is all stored in the target redundant memory area, and the to-be-kept storage data is all stored in an original hard disk write address.

8. The method of claim 1, wherein, Determining the to-be-transferred storage data according to the to-be-stored data comprises: determining the to-be-stored data as to-be-transferred storage data; and the to-be-stored data completing data storage, including that the to-be-transferred storage data is all stored to the target redundant memory area.

9. A processing device storing tasks, wherein, comprising: a slow node identification module, configured to identify a slow node in a plurality of storage nodes in a process in which the storage cluster stores data corresponding to a distributed storage task by the plurality of storage nodes on a hard disk; a data determination module, configured to determine to-be-stored data corresponding to a remaining storage task on the slow node, and determine to-be-transferred storage data according to the to-be-stored data; a memory writing module, configured to determine a target redundant memory area matched with the slow node according to a redundant memory topology table, and store the to-be-transferred storage data to the target redundant memory area; a message generation module, configured to generate a message that a storage task of the slow node is completed in response to the to-be-stored data completing data storage.

10. A computer readable storage medium, having stored thereon a computer program, the program being executed by a processor to implement the processing method of the storage task according to any one of claims 1 to 8.

11. An electronic device, comprising: comprising: one or more processors; a storage device, configured to store one or more programs, when the one or more programs are executed by the one or more processors, the one or more programs cause the one or more processors to implement the processing method of the storage task according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, the computer program being executed by a processor to implement the processing method of the storage task according to any one of claims 1 to 8.

Images