Distributed Data Processing Method, Apparatus, Device, Medium and Program Product

Abstract

The present application discloses a distributed data processing method, apparatus, device, medium and program product. The distributed data processing method of the present application includes: receiving a distributed data processing request and obtaining the identification information of the distributed data processing request; determining the request type of the distributed data processing request; determining a target access layer for processing the distributed data processing request among multiple access layers according to the identification information and the request type; performing an expansion process on the target access layer based on the scheduling information of the target access layer, and processing the distributed data processing request according to the processed target access layer. This solution can achieve the user's imperceptibility during the expansion of the distributed storage system, and at the same time improve the performance of the distributed storage system.

Description

Technical Field

[0001] This application relates to the technical field of cloud data processing, and in particular, to a distributed data processing method, apparatus, electronic device, computer storage medium, and computer program product. Background Art

[0002] Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that combines a large number of various types of storage devices (storage devices are also called storage nodes) in the network through functions such as cluster applications, grid technology, and distributed file systems, and collaborates through application software or application interfaces to jointly provide data storage and service access functions to the outside world.

[0003] With the explosive growth of user data and the increasing business requirements, the data volume of the storage cluster is also continuously growing. To meet the business growth needs, it is necessary to expand the original system. In the related art, the way to expand the original system is to expand the main node. This method requires preparing new node devices first, then making data backups, and migrating the data after the backup is completed to the prepared node devices one by one to achieve expansion.

[0004] However, in this expansion method, if network congestion occurs, it is very likely that data backup and migration will fail, resulting in expansion failure. Moreover, during the expansion process, due to the need to perform data backup and migration and other operations between each node (including the original main node and the newly prepared node), the message write request on the business side cannot be completed normally, affecting normal use. Summary of the Invention

[0005] The embodiments of this application provide a distributed data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can achieve user imperceptibility during the expansion of the distributed storage system and improve the performance of the distributed storage system at the same time.

[0006] In a first aspect, this application provides an embodiment of a distributed data processing method, including:

[0007] Receiving a distributed data processing request and obtaining the identification information of the distributed data processing request;

[0008] Determining the request type of the distributed data processing request;

[0009] Obtaining the target identification corresponding to the request type from the identification information;

[0010] Determining the storage location corresponding to the target identification;

[0011] Determine a target access layer for processing the distributed data processing request among multiple access layers based on a preset topology tool and the storage location;

[0012] When the target access layer fails, determine a temporary access layer among multiple access layers, and send the distributed data processing request to the temporary access layer; process the distributed data processing request according to the temporary access layer, and when communication is established between the temporary access layer and the target access layer, send the distributed data processing request from the temporary access layer to the target access layer;

[0013] When the capacity of the target access layer is insufficient, query the mapping relationship between the storage location and the target access layer based on the topology tool, expand the capacity of the target access layer, and process the distributed data processing request according to the processed target access layer.

[0014] In a second aspect, an embodiment of the present application further provides a distributed data processing device, including:

[0015] A receiving module, configured to receive a distributed data processing request and obtain identification information of the distributed data processing request;

[0016] A first determination module, configured to determine the request type of the distributed data processing request;

[0017] A second determination module, configured to obtain a target identifier corresponding to the request type from the identification information; determine the storage location corresponding to the target identifier; determine a target access layer for processing the distributed data processing request among multiple access layers based on a preset topology tool and the storage location;

[0018] A processing module, configured to when the target access layer fails, determine a temporary access layer among multiple access layers, and send the distributed data processing request to the temporary access layer; process the distributed data processing request according to the temporary access layer, and when communication is established between the temporary access layer and the target access layer, send the distributed data processing request from the temporary access layer to the target access layer; when the capacity of the target access layer is insufficient, query the mapping relationship between the storage location and the target access layer based on the topology tool, expand the capacity of the target access layer, and process the distributed data processing request according to the processed target access layer.

[0019] Optionally, in some embodiments of the present application, the processing module includes:

[0020] A first processing unit, configured to perform persistence processing on the distributed data processing request;

[0021] A determination unit, configured to determine a processing node corresponding to the target access layer, and determine a tape library group bound to the processing node based on a preset binding relationship;

[0022] A second processing unit, configured to process the persistently processed distributed data processing request according to the tape library group bound to the processing node.

[0023] Optionally, in some embodiments of the present application, the second processing unit is specifically configured to:

[0024] Determine a request type corresponding to the distributed data processing request;

[0025] When the request type corresponding to the distributed data processing request is a first type, obtain the disk capacity of the tape library group bound to the processing node; when the disk capacity of the tape library group bound to the processing node meets a preset condition, process the persistently processed distributed data processing request according to the tape library group bound to the processing node;

[0026] When the request type corresponding to the distributed data processing request is a second type, process the persistently processed distributed data processing request according to the grouping attribute corresponding to the distributed data processing request and the tape library group bound to the processing node.

[0027] Optionally, in some embodiments of the present application, the apparatus further includes a grouping module, and the grouping module is configured to:

[0028] Group a preset plurality of tape libraries according to a preset coding method to obtain a plurality of tape library groups each including the same number of tape libraries;

[0029] Bind the processing node corresponding to the access layer to the tape library group according to the number of tape libraries in the tape library group.

[0030] Optionally, in some embodiments of the present application, the apparatus further includes an expansion module, and the expansion module is configured to:

[0031] When the hardware corresponding to the tape library is expanded, expand the processing node corresponding to the tape library according to the expansion ratio of the hardware.

[0032] In a third aspect, an embodiment of the present application further provides an electronic device, including a processor and a memory, where the memory stores an application program, and the processor is configured to run the application program in the memory to implement the steps in the distributed data processing method provided in the embodiments of the present application.

[0033] Fourthly, an embodiment of the present application further provides a computer-readable storage medium, which stores multiple instructions adapted to be loaded by a processor to execute the steps in the distributed data processing method provided by the embodiment of the present application.

[0034] Fifthly, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which when executed by a processor implement the steps in the distributed data processing method provided by the embodiment of the present application.

[0035] In an embodiment of the present application, after receiving a distributed data processing request and obtaining the identification information of the distributed data processing request, the request type of the distributed data processing request is determined. Then, based on the identification information and the request type, a target access layer for processing the distributed data processing request is determined among multiple access layers. Finally, based on the scheduling information of the target access layer, the target access layer is extended, and the distributed data processing request is processed according to the processed target access layer. The distributed data processing solution provided by the present application, after receiving a distributed data processing request, flexibly selects the corresponding access layer according to its identification information and request type. When actually processing, the target access layer is extended according to the scheduling information of the access layer. The entire extension process does not require the distributed storage system to be shut down and is imperceptible to the user level. At the same time, it can be extended accordingly according to the scheduling information corresponding to different access layers, thereby improving the performance of the distributed storage system. Description of the Drawings

[0036] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present application. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0037] Figure 1 is a schematic diagram of the architecture of a distributed database system in the related art;

[0038] Figure 2 is a schematic diagram of the scenario of the distributed data processing method provided by the embodiment of the present application;

[0039] Figure 3 is a schematic flowchart of the distributed data processing method provided by the embodiment of the present application;

[0040] Figure 4 is a schematic diagram of the binding between a processing node and a tape library in the distributed data processing method provided by the embodiment of the present application;

[0041] Figure 5 It is a schematic flowchart of the data sedimentation process provided by an embodiment of the present application;

[0042] Figure 6 It is a schematic diagram for determining the storage location during the data sedimentation process provided by an embodiment of the present application;

[0043] Figure 7 It is a schematic flowchart of the data reheating process provided by an embodiment of the present application;

[0044] Figure 8 It is a schematic diagram for determining the storage location during the data reheating process provided by an embodiment of the present application;

[0045] Figure 9 It is a schematic flowchart of the data deletion process provided by an embodiment of the present application;

[0046] Figure 10 A schematic diagram for determining the storage location during the data deletion process provided by an embodiment of the present application;

[0047] Figure 11 It is a schematic structural diagram of a distributed data processing device provided by an embodiment of the present application;

[0048] Figure 12 It is another schematic structural diagram of a distributed data processing device provided by an embodiment of the present application;

[0049] Figure 13 It is yet another schematic structural diagram of a distributed data processing device provided by an embodiment of the present application;

[0050] Figure 14 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0051] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0052] The present application provides a distributed data processing method, device, electronic device, computer storage medium, and computer program product.

[0053] The distributed data processing can be specifically integrated into a server. Herein, the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, network acceleration services, and big data and artificial intelligence platforms. The server and the terminal can be directly or indirectly connected through wired or wireless communication means. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication means, which is not limited in this application.

[0054] For example, please refer to Figure 1 , this application provides a distributed data processing system, which includes an object storage (CloudObeject Storage, COS) database, and includes an access layer (Captain), a processing layer, a resource management layer (Icecenter), and a data storage engine (YottaStore). Among them, the access layer is used for: receiving and redistributing signaling, which depends on an external storage to persistently store the signaling (i.e., the distributed data processing request). When the access layer crashes or fails, since the signaling is persistently processed, the signaling will not be lost; in addition, in the embodiments of this application, a distributed data processing usually deploys multiple access layers to provide services externally. The processing layer includes multiple processing nodes (Iceworker), and the processing nodes are used to execute the tasks corresponding to the signaling, such as sedimentation, reheating, or deletion tasks. The resource management layer stores the resource information of all tape libraries and indirectly guides the processing nodes. For example, if a processing node needs to sediment data, it needs to apply for space from the resource management layer and can only write data to the tape library after the application. COS can perform data interaction by installing a COS plug-in in the application program. COS sends distributed data processing to the access layer, and the access layer sends dispatch tasks to multiple processing nodes according to the distributed data processing; each working stage obtains the data set corresponding to the data identification set from the corresponding tape library group according to the corresponding dispatch task and returns the data set to the data storage engine, thereby realizing distributed data processing.

[0055] The tape mentioned in this application is a low-cost storage medium. The tape library (TapeLibrary) is a hardware container that manages multiple tapes and can realize the reading and writing of data in the tapes. The tape library group (TapeLibraryGroup) is a collection of multiple tape libraries, which is also a hardware container that manages multiple tapes and can realize the reading and writing of data in the tapes.

[0056] The overall process is as follows: The user triggers a distributed data processing request through the network or application programming interface. The COS platform sends the distributed data processing request triggered by the user to the access layer. After receiving the distributed data processing request, the access layer performs persistent processing on the distributed data processing request, and then sends the distributed data processing request to the processing node after processing. The processing node performs hardware interaction with the tape library to execute specific tasks.

[0057] It should be noted that when the distributed data processing system receives a distributed data processing request, it can determine the target access layer for processing the distributed data processing request from multiple access layers according to the identification information and request type corresponding to the distributed data processing request. Finally, the target access layer and its corresponding processing node are used to process the distributed data processing request.

[0058] The distributed data processing solution provided by this application, after receiving a distributed data processing request, flexibly selects the corresponding access layer according to its identification information and request type. During actual processing, the target access layer is extended according to the scheduling information of the access layer. The entire extension process does not require the distributed storage system to be shut down and is imperceptible to the user. At the same time, it can be extended accordingly according to the scheduling information corresponding to different access layers, thereby improving the performance of the distributed storage system.

[0059] For example, referring to Figure 2 , taking the case where the distributed data processing device is integrated in the server as an example, after the server receives the distributed data processing request and obtains the identification information of the distributed data processing request, it determines the request type of the distributed data processing request. Then, according to the identification information and request type, it determines the target access layer for processing the distributed data processing request from multiple access layers. Finally, based on the scheduling information of the target access layer, the target access layer is extended and processed, and the distributed data processing request is processed according to the processed target access layer.

[0060] Among them, it can be understood that in the specific implementation manner of this application, related data such as attribute data, attribute sets, and attribute subsets are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent needs to be obtained, and the collection, use, and processing of related data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.

[0061] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0062] This embodiment will be described from the perspective of the distributed data processing device. The distributed data processing device can be specifically integrated in an electronic device, and the electronic device can be a server.

[0063] An embodiment of the present application provides a distributed data processing method, including: receiving a distributed data processing request and obtaining identification information of the distributed data processing request; determining a request type of the distributed data processing request; determining a target access layer for processing the distributed data processing request from multiple access layers according to the identification information and the request type; performing an expansion process on the target access layer based on scheduling information of the target access layer, and processing the distributed data processing request according to the processed target access layer.

[0064] Please refer to Figure 3 , Figure 3 , which is a schematic flowchart of the road network processing method provided by the present application. The specific process of the road network processing method can be as follows:

[0065] 101. Receive a distributed data processing request and obtain identification information of the distributed data processing request.

[0066] For example, specifically, a distributed data processing request can be received through a network or an application programming interface. The distributed data processing request can include a data settlement request, a data reheating request, and a data deletion request.

[0067] Among them, data settlement means that hot data often has its heat increased due to being accessed a large number of times in a short period after being uploaded, and after a period of time, its heat gradually decreases or it is no longer necessary to be accessed in real time. The process of converting these hot data into low-frequency storage or archival storage. Data reheating means that data in a cold data storage engine (tape medium) is retransmitted to a hot data storage engine for storage, and the corresponding data in the cold data storage engine is deleted.

[0068] In addition, the distributed data processing request carries the user's unique resource identifier (APPid), the storage bucket identifier (Bucketid), the data block identifier (Blockid) corresponding to the data to be processed, and the tape library group identifier (Groupid).

[0069] 102. Determine the request type of the distributed data processing request.

[0070] It can be understood that the distributed data processing request can include a data settlement request, a data reheating request, and a data deletion request, that is, the request type of the distributed data processing request can include a data settlement type, a data reheating type, and a data deletion type. When performing distributed data processing subsequently, it is necessary to process according to different request types.

[0071] 103. Determine a target access layer for processing the distributed data processing request from multiple access layers according to the identification information and the request type.

[0072] The distributed data processing request can be a data settlement request, a data reheat request, or a data deletion request. In the embodiments of the present application, a multi-dimensional processing strategy is adopted, that is, different methods are used for different distributed data processing requests. Specifically, different distributed data processing requests use different identifiers subsequently. For example, the unique resource identifier (APPid) and the bucket identifier (Bucketid) are used for the data settlement request, the tape library group identifier (Groupid) is used for the data reheat request, and the data block identifier (Blockid) is used for the data deletion request. Therefore, optionally, in some embodiments of the present application, the step of "determining a target access layer for processing the distributed data processing request among multiple access layers according to the identifier information and the request type" may specifically include:

[0073] Obtain the target identifier corresponding to the request type from the identifier information;

[0074] Determine a target access layer for processing the distributed data processing request among multiple access layers according to the target identifier.

[0075] For example, specifically, for processing the data settlement request (ArchiveTasks), when Captain processes the settlement request, a linear expansion is performed in the way of SlotHash (a hash processing algorithm). Specifically, a long integer string (Slotseed) is calculated by using the unique resource identifier (APPid) and the bucket identifier (Bucketid) in the distributed data processing request. Then, the corresponding storage location (SlotLocation) is determined through the Slotseed. Finally, according to the storage location and the preset topology tool, a target access layer for processing the distributed data processing request is determined among multiple access layers. That is, optionally, in some embodiments of the present application, the step of "determining a target access layer for processing the distributed data processing request among multiple access layers according to the target identifier" may specifically include:

[0076] Determine the storage location corresponding to the target identifier;

[0077] Determine a target access layer for processing the distributed data processing request among multiple access layers based on the preset topology tool and the storage location.

[0078] For example, specifically, still taking the processing of a data settlement request as an example, after obtaining a long integer string (Slotseed), the preset position calculation formula SlotLocation = SlotSeed % SlotSize is obtained, where SlotSize is a business-customized value, usually not exceeding 65545. Finally, based on this storage location (SlotLocation) and the preset topology tool (Cayman), the target access layer for processing distributed data processing requests is determined among multiple access layers. Specifically, the topology tool (Cayman) is used to query and determine the access layer identifier of the target access layer corresponding to the storage location (SlotLocation), thereby determining the target access layer for processing distributed data processing requests. It should be noted that the topology tool (Cayman) is used to manage the mapping relationship between Slots (storage slots or storage subspaces) and Shards (fragments); Shard is a concept in Erasure Coding (EC), and EC is a coding fault tolerance technology, which was first used in the communication industry to solve the problem of partial data loss during transmission. Its basic principle is to segment the transmitted signal, add certain checksums, and then make the segments correlate with each other. Even if some signals are lost during transmission, the receiving end can still calculate the complete information through algorithms. In data storage, erasure coding divides data into segments, expands and encodes redundant data blocks, and stores them in different locations, such as disks, storage nodes, or other geographical locations. One column in EC is a Shard. For example, when the EC coding is KN, when a Block is finally written to the tape, it is N Shards, including K data Shards and (N - K) parity Shards. That is, in the tape library, data is stored in units of Blocks. A Block can be sharded to obtain multiple data shards (Shards), and the multiple data shards are written into multiple cache libraries respectively, and then written from the cache libraries to the tape on the tape library. When a user initiates a data settlement request through a web page or application on the client side, the unique resource identifier (APPid) and storage bucket identifier (Bucketid) of the data settlement request are calculated, the storage location (SlotLocation) corresponding to the data settlement request is calculated, and finally, the access layer identifier of the target access layer corresponding to this storage location (SlotLocation) is determined through Cayman, thereby completing the determination of the target access layer for processing distributed data processing requests among multiple access layers.

[0079] 104. Based on the scheduling information of the target access layer, perform an expansion process on the target access layer, and process the distributed data processing request according to the processed target access layer.

[0080] The scheduling information of the target access layer may include fault data and capacity data of the target access layer. The fault data may be used to indicate whether the target access layer is in a fault state; and the capacity data may indicate the current capacity of the target access layer.

[0081] It is understandable that in the actual distributed data processing process, access layer failure or insufficient access layer capacity often occurs. The embodiments of the present application have made corresponding processing for these two situations, which are as follows:

[0082] When a target access layer fails, the embodiment of the present application sends a request of the target access layer to other access layers for acceptance until the target access layer failure is repaired. That is, optionally, in some embodiments of the present application, the step of "based on the scheduling information of the target access layer, performing extended processing on the target access layer, and processing the distributed data processing request according to the processed target access layer" may specifically include:

[0083] When a target access layer fails, a temporary access layer is determined among multiple access layers, and a distributed data processing request is sent to the temporary access layer;

[0084] The distributed data processing request is processed according to the temporary access layer, and when the temporary access layer establishes communication with the target access layer, the distributed data processing request is sent from the temporary access layer to the target access layer.

[0085] For example, specifically, when access layer A fails, the access layer management (CaptainClient) can temporarily send the distributed data processing request to other access layers for processing, such as to access layer B for processing; when access layer B takes over the distributed data processing request of access layer A, it regularly establishes communication with access layer A. After the communication is established, the temporarily carried distributed data processing request is returned, that is, access layer B sends the distributed data processing request to access layer A.

[0086] When the capacity of the target access layer is insufficient, shard expansion may be performed, that is, optionally, in some embodiments of the present application, the step of "processing the distributed data processing request according to the processed target access layer" may specifically include:

[0087] When the capacity of the target access layer is insufficient, the mapping relationship between the storage location and the target access layer is queried based on the preset topology tool, and the capacity of the target access layer is expanded.

[0088] Suppose an access layer is composed of Node 1, Node 2, and Node 3, which can be three ports on one machine or three different servers. Then, if the hash slot method is used to allocate 16384 slots, the slot ranges borne by the three nodes are as follows:

[0089] Node 1 covers 0 - 5460;

[0090] Node 2 covers 5461 - 10922;

[0091] Node 3 covers 10923 - 16383.

[0092] Using the sharding expansion method, that is, adding new Sharding (fragments) to the cluster, then the cluster only needs to move some slots in Nodes 1, 2, and 3 to Node 4. For example, the following operations can be performed:

[0093] Node 1 covers 1365 - 5460;

[0094] Node 2 covers 6827 - 10922;

[0095] Node 3 covers 12288 - 16383;

[0096] Node 4 covers 0 - 1364, 5461 - 6826, 10923 - 12287.

[0097] Furthermore, during actual processing, to avoid the loss of distributed data processing requests caused by access layer failures, it is necessary to process the distributed data processing requests. Then, according to the processing node of the target access layer, process the persistently processed distributed data processing requests. That is, optionally, in some embodiments of the present application, the step of "processing the distributed data processing request according to the processed target access layer" may specifically include:

[0098] Perform persistent processing on the distributed data processing request;

[0099] Determine the processing node corresponding to the target access layer, and based on the preset binding relationship, determine the tape library group bound by the processing node;

[0100] Process the persistently processed distributed data processing request according to the tape library group bound by the processing node.

[0101] For example, taking the data warm-up request as an example, data is stored in the tape library in units of blocks (Blocks). A Block can be sharded to obtain multiple data shards, and the multiple data shards are written into multiple cache libraries respectively, and then written from the cache library to the tape on the tape library. The user initiates a warm-up request through a web page or an application at the client; COS sends the user's warm-up request to Captain; Captain temporarily caches the received user request in Buffer and accumulates as many warm-up requests as possible within the warm-up timeout period: when there are enough warm-up requests or the warm-up request has timed out, Captain sends the data to IceWorker for the execution of the warm-up request; after IceWorker warms up the data from the tape library, it writes the data back to YottaStore and notifies the user that the warm-up is successful.

[0102] It should be noted that the tape library is a batch job system. Captain accumulates enough requests before distributing them to avoid the processing node IceWorker warming up only a small amount of data at a time.

[0103] For the data settlement request, before the processing node IceWorker applies for a task from Captain, it needs to apply for a quota from the resource management layer IceCenter first. If the tape library where IceWorker is located is full, it will no longer continue to apply for resources from Captain; in the process of processing this data settlement request, the target access layer (Captain) does not distinguish the distributed data settlement requests sent. That is, whichever processing node (Iceworker) in which tape library group (Group) applies for the task, the task will be sent to that processing node (Iceworker).

[0104] However, it is different for the data warm-up task and the data deletion task. Because the data warm-up request in the data warm-up task carries the attribute of the tape library group (Group), the processing unit only reads the data warm-up requests of the tape library group (Group) to which it belongs; the data deletion request is similar to the data warm-up request, and the data deletion request carries the attribute of the tape library group (Group). Therefore, the processing unit only reads the data deletion requests of the tape library group (Group) to which it belongs.

[0105] Therefore, it can be understood that for distributed data requests of different request types, this application adopts different processing methods. That is, optionally, in some embodiments of this application, the step of "processing the distributed data processing request subjected to persistent processing according to the tape library group bound by the processing node" may specifically include:

[0106] Determine the request type corresponding to the distributed data processing request;

[0107] When the request type corresponding to the distributed data processing request is the first type, obtain the disk capacity of the tape library group bound to the processing node; when the disk capacity of the tape library group bound to the processing node meets the preset condition, process the persistently processed distributed data processing request according to the tape library group bound to the processing node;

[0108] When the request type corresponding to the distributed data processing request is the second type, process the persistently processed distributed data processing request according to the grouping attribute corresponding to the distributed data processing request and the tape library group bound to the processing node.

[0109] In the embodiments of the present application, the data settlement request type is determined as the first type, and the data reheating request type and the data deletion type are determined as the second type; the preset condition may be that the capacity of the tape library where the processing node is located is greater than or equal to the data volume corresponding to the distributed data processing request.

[0110] In addition, it should be noted that in the embodiments of the present application, tape storage supports EC encoding. Therefore, the tape libraries can be grouped according to the EC encoding method. Taking the EC encoding of KxNy as an example, where both K and N are positive integers, as Figure 4 shown, the number of tape libraries that a tape library group needs to contain is N, and a tape library group corresponds to all processing nodes of a processing layer. That is, optionally, in some embodiments of the present application, the distributed data processing method of the present application may specifically further include:

[0111] Group a preset number of tape libraries according to a preset encoding method to obtain multiple tape library groups containing the same number of tape libraries;

[0112] Bind the processing nodes corresponding to the access layer to the tape library group according to the number of tape libraries in the tape library group.

[0113] As can be seen from the foregoing, the data of K columns of processing nodes is the original data, and the parity data is the data of K - N columns. If the number of incorrect columns returned ≤ (N - K), the processing node can calculate the original data by EC. Then, in the scenario of a data reheating request, it indicates that this data reheating request can be successfully executed.

[0114] Optionally, in some embodiments of the present application, when the hardware corresponding to the tape library is expanded, expand the processing nodes corresponding to the tape library according to the expansion ratio of the hardware.

[0115] The above is the distributed data processing flow of the embodiments of the present application.

[0116] The embodiment of the present application provides a distributed data processing method. After receiving a distributed data processing request, the corresponding access layer is flexibly selected according to its identification information and request type. When performing actual processing, the target access layer is expanded according to the scheduling information of the access layer. The entire expansion process does not require the distributed storage system to be shut down, and is imperceptible to the user level. At the same time, corresponding expansion can be performed according to the scheduling information corresponding to different access layers, thereby improving the performance of the distributed storage system.

[0117] In order to facilitate the understanding of the distributed data processing solution of the present application, the following three scenarios are respectively described in detail, namely, data sedimentation, data recovery and data deletion.

[0118] See also Figure 5 , and combined with Figure 1 In the process of data sedimentation, when the target access layer (Captain) processes the data sedimentation request, SlotHash (a hash processing algorithm) is used for linear expansion. a. The CaptainClient side calculates a long integer string SlotSeed according to the unique resource identifier (APPid) and the bucket identifier (Bucketid) in the user request; then, locates the storage slot (Slot): SlotLocation = SlotSeed% SlotSize, where SlotSize is a business-defined value, usually not exceeding 65545; according to the storage slot (Slot) location, the CaptainId is obtained by looking up the table in Cayman. At this point, the CaptainClient can determine which Captain to send the sedimentation request to for task processing, such as Figure 6 shown.

[0119] Specifically, after receiving a data settlement request initiated by a user, the cold data storage engine (YottaStore) sends the data settlement request to the target access layer (Captaion). The target access layer (Captaion) performs clustering and persistence processing on the received data settlement request. Subsequently, it sends the clustered and persisted data settlement request to the processing node (IceWorker). The processing node (IceWorker) downloads data from the cold data storage engine (YottaStore) and slices the downloaded data (Block) according to EC. The data is written to the tape library in units of data blocks (Block). The Block is sliced according to EC, and the sliced data is Shard. Taking the EC encoding with the code KN as an example, KN means that K columns are data columns and (N - K) are parity columns, and the total number of data columns and parity columns is N columns; one column in EC is one Shard; first, the Block is cut into K equal-sized data Shards, and then (N - K) parity Shards are calculated. A total of N column Shards need to be written to the tape library. These N column Shards will be written to N different tape libraries for disaster recovery. The data settlement task is to synchronously write each data shard in the target data block to the tapes in different tape libraries, and one data shard corresponds to one tape library. Each tape library corresponds to a cache library. Before each data shard is written to the tape, the data shard is first written to the cache library, and then written from the cache library to the tape in the tape library. After the data shard is successfully written to the tape, the data in the cache library is cleared, that is, the cache is cleared. Among them, when the target access layer (Captain) fails, multiple access layers (Captain), that is, the remaining access layers, can autonomously degrade. When a certain Captain undertakes the request of the failed Captain, it communicates with the failed Captain (i.e., the target access layer) regularly. After the communication is successful, the temporarily carried request is returned. In addition, when the capacity of the target access layer (Captain) is insufficient, sharding expansion can be performed. After the expansion, the mapping relationship of SlotLocation<>CaptainID in Cayman is updated.

[0120] It should be noted that since the addressing time of tape storage is very long, in order to avoid the data being scattered in various places in the tape library during data warm-up, a clustering operation will be performed according to the user's data characteristics, so that the data with the same characteristics can be processed by the processing node (IceWorker) as much as possible, and then continuously written to the tape to achieve local continuity. In this way, when a certain user's data is warmed up, the user's data is likely to be locally continuous, which can significantly improve the warm-up performance.

[0121] Please refer to Figure 7 , and in combination with Figure 2, the data reheating process is similar to the data sedimentation process. Similarly, after the cold data storage engine (YottaStore) receives the data reheating request initiated by the user, it sends the data reheating request to the target access layer (Captaion). The target access layer (Captaion) clusters, persists, and statically processes the received data reheating request. Subsequently, it sends the clustered, persisted, and statically processed data reheating request to the processing node (IceWorker). The processing node (IceWorker) downloads the data from the cold data storage engine (YottaStore), slices the downloaded data (Block) according to EC, writes multiple data shards into multiple cache libraries respectively, and then writes them onto the tape in the tape library. The user initiates a reheating request through a web page or application on the client side; the user's reheating request can be sent to Captain; Captain temporarily caches the received user request in Buffer and accumulates as many reheating requests as possible within the reheating timeout period: when there are enough reheating requests or the reheating request has timed out, Captain sends the data to IceWorker for execution of the reheating request; after IceWorker reheats the data from the tape library, it writes the data back to YottaStore and notifies the user that the reheating is successful. Since the tape library is a batch job system, the purpose of static processing is to allow Captain to accumulate enough requests before distribution, thus avoiding IceWorker reheating only a small amount of data each time.

[0122] The difference is that when the target access layer (Captain) processes the data reheating request, it uses the SlotHash method for linear expansion. The SlotSeed is no longer the unique resource identifier (APPid) and the storage bucket identifier (Bucketid), but the tape library group identifier (GroupId), as Figure 8 shown.

[0123] Please refer to Figure 9 , and in combination with Figure 2 , the data deletion process is relatively simple. After the cold data storage engine (YottaStore) receives the data deletion request initiated by the user, it sends the data reheating request to the target access layer (Captaion). The target access layer (Captaion) aggregates, persists, and statically processes the received data reheating request. Subsequently, it sends the aggregated, persisted, and statically processed data deletion request to the processing node (IceWorker), and the processing node (IceWorker) performs the deletion operation.

[0124] It should be noted that deletion at the granularity of data blocks (Blocks) is performed during the aggregation of the target access layer (Captain). When the target access layer (Captain) receives a data deletion request for an object, it first determines whether the data size of the object to be deleted meets a preset value; if not, the data deletion request will be persisted until all the data blocks corresponding to the object receive the deletion execution. Then, the target access layer (Captain) will distribute a Block-level deletion task to the processing node (IceWorker); after the processing node (IceWorker) initiates the deletion to the tape library, the tape library will delete the file corresponding to this Block from the file system.

[0125] It should also be noted that when the target access layer (Captain) processes data deletion requests, it uses the SlotHash method for linear expansion. The SlotSeed is no longer the unique resource identifier (APPid) and the bucket identifier (Bucketid), but the data block identifier (BlockId), as Figure 10 shown.

[0126] To better implement the above method, an embodiment of the present application also provides a distributed data processing device, as Figure 11 shown. The distributed data processing device may include: a receiving module 301, a first determination module 302, a second determination module 303, and a processing module 304, as follows:

[0127] The receiving module 301 is configured to receive a distributed data processing request and obtain the identification information of the distributed data processing request.

[0128] For example, specifically, a distributed data processing request can be received through a network or an application programming interface. The distributed data processing request may include a data settlement request, a data reheating request, and a data deletion request.

[0129] The first determination module 302 is configured to determine the request type of the distributed data processing request.

[0130] The request type of the distributed data processing request may include a data settlement type, a data reheating type, and a data deletion type. When performing distributed data processing subsequently, it is necessary to process according to different request types.

[0131] The second determination module 303 is configured to determine the target access layer for processing the distributed data processing request in multiple access layers according to the identification information and the request type.

[0132] The distributed data processing requests can be data settlement requests, data reheating requests, or data deletion requests. In the embodiments of this application, a multi-dimensional processing strategy is adopted, that is, different methods are used for different distributed data processing requests. Specifically, different distributed data processing requests use different identifiers in the subsequent process. For example, for data settlement requests, the unique resource identifier (APPid) and the bucket identifier (Bucketid) are used. For data reheating requests, the tape library group identifier (Groupid) is used. For data deletion requests, the data block identifier (Blockid) is used.

[0133] Optionally, in some embodiments of this application, the second determination module 303 may specifically include:

[0134] An acquisition unit, configured to acquire the target identifier corresponding to the request type from the identification information;

[0135] A third processing unit, configured to determine a target access layer for processing the distributed data processing request from multiple access layers according to the target identifier.

[0136] Optionally, in some embodiments of this application, the third processing unit may specifically be configured to: determine the storage location corresponding to the target identifier; and determine a target access layer for processing the distributed data processing request from multiple access layers based on a preset topology tool and the storage location.

[0137] A processing module 304, configured to perform expansion processing on the target access layer based on the scheduling information of the target access layer, and process the distributed data processing request according to the processed target access layer.

[0138] Among them, the scheduling information of the target access layer may include the failure data and capacity data of the target access layer. The failure data may be used to indicate whether the target access layer is in a failure state; the capacity data may indicate the current capacity of the target access layer.

[0139] It can be understood that in the actual process of distributed data processing, situations such as access layer failures or insufficient access layer capacity often occur. Optionally, in some embodiments of this application, the processing module 304 may specifically be configured to:

[0140] Send the distributed data processing request to the temporary access layer;

[0141] Process the distributed data processing request according to the temporary access layer. When the temporary access layer establishes communication with the target access layer, send the distributed data processing request from the temporary access layer to the target access layer.

[0142] Optionally, in some embodiments of this application, the processing module 304 may specifically be configured to:

[0143] When the capacity of the target access layer is insufficient, query the mapping relationship between the storage location and the target access layer based on a preset topology tool, and expand the capacity of the target access layer.

[0144] Optionally, in some embodiments of the present application, the processing module 304 may specifically include:

[0145] A first processing unit for performing persistent processing on the distributed data processing request;

[0146] A determination unit for determining the processing node corresponding to the target access layer, and determining the tape library group bound to the processing node based on a preset binding relationship;

[0147] A second processing unit for processing the persistent distributed data processing request according to the tape library group bound to the processing node.

[0148] Optionally, in some embodiments of the present application, the second processing unit may specifically be used for:

[0149] Determining the request type corresponding to the distributed data processing request;

[0150] When the request type corresponding to the distributed data processing request is the first type, obtain the disk capacity of the tape library group bound to the processing node; when the disk capacity of the tape library group bound to the processing node meets the preset conditions, process the persistent distributed data processing request according to the tape library group bound to the processing node;

[0151] When the request type corresponding to the distributed data processing request is the second type, process the persistent distributed data processing request according to the grouping attribute corresponding to the distributed data processing request and the tape library group bound to the processing node.

[0152] Optionally, in some embodiments of the present application, please refer to Figure 12 , the distributed data processing device of the present application further includes a grouping module 305, and the grouping module 305 may specifically be used for: grouping a preset plurality of tape libraries according to a preset coding method to obtain a plurality of tape library groups each containing the same number of tape libraries; binding the processing node corresponding to the access layer to the tape library group according to the number of tape libraries in the tape library group.

[0153] Optionally, in some embodiments of the present application, please refer to Figure 13 , the distributed data processing device of the present application further includes an expansion module 306, and the expansion module 306 may specifically be used for: when the hardware corresponding to the tape library is expanded, expanding the processing node corresponding to the tape library according to the expansion ratio of the hardware.

[0154] As can be seen from the above, the embodiment of the present application provides a distributed data processing device. After the receiving module 301 receives a distributed data processing request and obtains the identification information of the distributed data processing request, the first determination module 302 determines the request type of the distributed data processing request. Then, the second determination module 303 determines the target access layer for processing the distributed data processing request from multiple access layers according to the identification information and the request type. Finally, the processing module 304 performs extended processing on the target access layer based on the scheduling information of the target access layer, and processes the distributed data processing request according to the processed target access layer. In the distributed data processing solution provided by the present application, after receiving a distributed data processing request, the corresponding access layer is flexibly selected according to its identification information and request type. During actual processing, the target access layer is extended according to the scheduling information of the access layer. The entire extension process does not require the distributed storage system to be shut down and is imperceptible to the user. At the same time, corresponding extensions can be made according to the scheduling information corresponding to different access layers, thereby improving the performance of the distributed storage system.

[0155] The embodiment of the present application also provides an electronic device, as Figure 14 shown, which shows the structural schematic diagram of the electronic device involved in the embodiment of the present application. Specifically:

[0156] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art can understand that Figure 14 the structure of the electronic device shown in

[0157] does not constitute a limitation on the electronic device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0158] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and distributed data processing by running the software programs and modules stored in the memory 402. The memory 402 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 402 can also include a memory controller to provide the processor 401 with access to the memory 402.

[0159] The electronic device further includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0160] The electronic device may further include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0161] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402 to implement various functions as follows:

[0162] Receive a distributed data processing request, and obtain the identification information of the distributed data processing request; determine the request type of the distributed data processing request; determine a target access layer for processing the distributed data processing request among multiple access layers according to the identification information and the request type; based on the scheduling information of the target access layer, perform an expansion process on the target access layer, and process the distributed data processing request according to the processed target access layer.

[0163] Optionally, in some embodiments of the present application, when performing an expansion process on the target access layer based on the scheduling information of the target access layer and processing the distributed data processing request according to the processed target access layer, the processor 401 is specifically used for:

[0164] When there is a fault in the target access layer, a temporary access layer is determined among multiple access layers, and a distributed data processing request is sent to the temporary access layer;

[0165] Process the distributed data processing request according to the temporary access layer. When communication is established between the temporary access layer and the target access layer, send the distributed data processing request from the temporary access layer to the target access layer.

[0166] Optionally, in some embodiments of the present application, when processing the distributed data processing request according to the processed target access layer, the processor 401 is specifically configured to:

[0167] When the capacity of the target access layer is insufficient, query the mapping relationship between the storage location and the target access layer based on a preset topology tool, and expand the capacity of the target access layer.

[0168] Optionally, in some embodiments of the present application, when processing the distributed data processing request according to the processed target access layer, the processor 401 is specifically configured to:

[0169] Perform persistent processing on the distributed data processing request;

[0170] Determine the processing node corresponding to the target access layer, and based on a preset binding relationship, determine the tape library group bound to the processing node;

[0171] Process the persistently processed distributed data processing request according to the tape library group bound to the processing node.

[0172] Optionally, in some embodiments of the present application, when processing the persistently processed distributed data processing request according to the tape library group bound to the processing node, the processor 401 is specifically configured to:

[0173] Determine the request type corresponding to the distributed data processing request;

[0174] When the request type corresponding to the distributed data processing request is the first type, obtain the disk capacity of the tape library group bound to the processing node; when the disk capacity of the tape library group bound to the processing node meets the preset condition, process the persistently processed distributed data processing request according to the tape library group bound to the processing node;

[0175] When the request type corresponding to the distributed data processing request is the second type, process the persistently processed distributed data processing request according to the grouping attribute corresponding to the distributed data processing request and the tape library group bound to the processing node.

[0176] Optionally, in some embodiments of the present application, the processor 401 is further configured to:

[0177] Group a preset plurality of tape libraries according to a preset encoding method to obtain a plurality of tape library groups each containing the same number of tape libraries;

[0178] Bind the processing nodes corresponding to the access layer to the tape library groups according to the number of tape libraries in the tape library groups.

[0179] Optionally, in some embodiments of the present application, the processor 401 is specifically further configured to:

[0180] When the hardware corresponding to the tape library is expanded, expand the processing nodes corresponding to the tape library according to the expansion ratio of the hardware.

[0181] Optionally, in some embodiments of the present application, when determining a target access layer for processing a distributed data processing request among multiple access layers according to the identification information and the request type, the processor 401 is specifically configured to: obtain a target identifier corresponding to the request type from the identification information; determine, according to the target identifier, a target access layer for processing the distributed data processing request among the multiple access layers.

[0182] Optionally, in some embodiments of the present application, when determining a target access layer for processing a distributed data processing request among multiple access layers according to the target identifier, the processor 401 is specifically configured to: determine a storage location corresponding to the target identifier; determine, based on a preset topology tool and the storage location, a target access layer for processing the distributed data processing request among the multiple access layers.

[0183] The electronic device provided by the embodiments of the present application, after receiving a distributed data processing request, flexibly selects a corresponding access layer according to its identification information and the request type, and during actual processing, expands the target access layer according to the scheduling information of the access layer. The entire expansion process does not require the distributed storage system to be shut down and is imperceptible to the user. At the same time, it can perform corresponding expansion according to the scheduling information corresponding to different access layers, thereby improving the performance of the distributed storage system.

[0184] For the specific implementation of each of the above operations, reference may be made to the previous embodiments and will not be elaborated herein.

[0185] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0186] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the distributed data processing methods provided in the embodiments of the present application, the beneficial effects achievable by any of the distributed data processing methods provided in the embodiments of the present application can be realized. For details, refer to the previous embodiments and will not be elaborated here.

[0187] Wherein, according to one aspect of the present application, a computer program product or a computer program is provided. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various alternative implementations of the above-mentioned distributed data processing aspect.

[0188] The above has introduced in detail a distributed data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A distributed data processing method, characterized in that: include: Receiving a distributed data processing request and obtaining identification information of the distributed data processing request; Determining a request type of the distributed data processing request; Acquire a target identifier corresponding to the request type from the identifier information; Determining a storage location corresponding to the target identifier; Based on a preset topology tool and the storage location, determining a target access layer for processing the distributed data processing request among multiple access layers; When the target access layer fails, a temporary access layer is determined among the multiple access layers, and the distributed data processing request is sent to the temporary access layer; Processing the distributed data processing request according to the temporary access layer, and sending the distributed data processing request from the temporary access layer to the target access layer when the temporary access layer establishes communication with the target access layer; When the capacity of the target access layer is insufficient, the mapping relationship between the storage location and the target access layer is queried based on the topology tool, the target access layer is expanded, and the distributed data processing request is processed according to the processed target access layer.

2. The distributed data processing method according to claim 1, characterized in that: The processing of the distributed data processing request according to the processed target access layer includes: Performing persistent processing on the distributed data processing request; Determine a processing node corresponding to the target access layer, and determine a tape library group bound to the processing node based on a preset binding relationship; The distributed data processing request for persistent processing is processed according to the tape library group bound to the processing node.

3. The distributed data processing method according to claim 2, characterized in that: The processing of the distributed data processing request for persistent processing according to the tape library group bound to the processing node includes: Determining a request type corresponding to the distributed data processing request; When the request type corresponding to the distributed data processing request is the first type, the disk capacity of the tape library group bound to the processing node is obtained; when the disk capacity of the tape library group bound to the processing node meets the preset condition, the distributed data processing request for persistent processing is processed according to the tape library group bound to the processing node; When the request type corresponding to the distributed data processing request is the second type, the distributed data processing request for persistent processing is processed according to the grouping attribute corresponding to the distributed data processing request and the tape library group bound to the processing node.

4. The distributed data processing method according to claim 2, characterized in that: Also includes: Grouping a plurality of preset tape libraries according to a preset coding method to obtain a plurality of tape library groups including the same number of tape libraries; According to the number of tape libraries in the tape library group, a processing node corresponding to the access layer is bound to the tape library group.

5. The distributed data processing method according to claim 2, characterized in that: Also includes: When the hardware corresponding to the tape library is expanded, the processing node corresponding to the tape library is expanded according to the expansion ratio of the hardware.

6. A distributed data processing device, characterized in that: include: A receiving module, used to receive a distributed data processing request and obtain identification information of the distributed data processing request; A first determining module, used to determine the request type of the distributed data processing request; A second determination module, configured to obtain a target identifier corresponding to the request type from the identifier information; Determining a storage location corresponding to the target identifier; Based on a preset topology tool and the storage location, determining a target access layer for processing the distributed data processing request among multiple access layers; A processing module, configured to determine a temporary access layer among multiple access layers when a fault occurs in the target access layer, and send the distributed data processing request to the temporary access layer; The distributed data processing request is processed according to the temporary access layer. When the temporary access layer establishes communication with the target access layer, the distributed data processing request is sent from the temporary access layer to the target access layer. When the capacity of the target access layer is insufficient, the mapping relationship between the storage location and the target access layer is queried based on the topology tool, the capacity of the target access layer is expanded, and the distributed data processing request is processed according to the processed target access layer.

7. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the distributed data processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the distributed data processing method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the distributed data processing method according to any one of claims 1 to 5 are implemented.

Images