Cross-storage cluster object copying method and device, electronic equipment and storage medium

Abstract

The invention discloses a cross-storage-cluster object copying method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-executed first copying task under the condition that a current node is a task control node of a first storage cluster, the first copying task can be used for copying the first data object set of the first storage cluster into the second storage cluster; based on the task type of the first copy task, the first copy task can be split into a group of copy subtasks, so that the object copy accuracy can be improved, and unnecessary resource waste is avoided; and distributing the group of copy sub-tasks to a plurality of task execution nodes of the first storage cluster, so that the task execution nodes in the plurality of task execution nodes copy the corresponding first data object into the second storage cluster according to the copy sub-tasks of the group of distributed copy sub-tasks, thereby realizing parallel processing of data object copying. And the object copying efficiency of the cross-storage cluster is further improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a method and device for copying objects across storage clusters, an electronic device, and a storage medium. Background Art

[0002] A single user can use multiple independent storage clusters for data object storage, leading to a large number of data copy operations between cluster systems. The cross-storage cluster object copy method used in related technologies involves the upper-layer business system using a standard interface to read data from one storage cluster and then write it to a second storage cluster.

[0003] However, the above-mentioned object copy method across storage clusters has a problem of low data copy efficiency due to multiple data copy interactions. Summary of the Invention

[0004] The present application provides a method and apparatus for copying objects across storage clusters, an electronic device, and a storage medium, to at least solve the problem of low data object copying efficiency in copying methods across object storage clusters in related technologies.

[0005] The present application provides an object copy method across storage clusters, comprising: when the current node is a task control node of a first storage cluster, obtaining a first copy task to be executed, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster; based on the task type of the first copy task, splitting the first copy task into a group of copy subtasks, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy part of the first data objects in the first data object set to the second storage cluster; distributing the group of copy subtasks to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks in the distributed group of copy subtasks.

[0006] The present application also provides an object copying device across storage clusters, comprising: a first acquisition unit, for acquiring a first copy task to be executed when the current node is a task control node of the first storage cluster, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster; a first splitting unit, for splitting the first copy task into a group of first copy subtasks based on the task type of the first copy task, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy part of the first data objects in the first data object set to the second storage cluster; a first execution unit, for distributing the group of copy subtasks to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks in the distributed group of copy subtasks.

[0007] In an exemplary embodiment, the task information of the copy task issued by the service platform to the first storage cluster is stored in the first database of the first storage cluster.

[0008] The above-mentioned first acquisition unit includes: a first search module, which is used to search for copy tasks whose task status belongs to a specified state set from the first database when the current node is the task control node of the first storage cluster, wherein the specified state set includes an unexecuted state and an incomplete split state, and the incomplete split state is a state in which the corresponding copy task has begun to be split into copy subtasks but the splitting has not been completed; a first determination module, which is used to, when a copy task whose task status belongs to the specified state set is searched, determine the copy task whose task status belongs to the specified state set and has the earliest creation time as the first copy task to be executed, wherein, when the task status of the first copy task is the unexecuted state, the task status of the first copy task is modified to the incomplete split state.

[0009] In an exemplary embodiment, the above-mentioned first splitting unit includes: a second determination module, which is used to determine the current split position based on the task mark of the first copy task when the task status of the first copy task is the incomplete split state, wherein the task mark of the first copy task is used to indicate the object position of the first first data object in the first data object set that has not been split into the copy subtask; a first splitting module, which is used to split the first data object in the first data object set that is located after the current split position into the copy subtask based on the task type of the first copy task, to obtain the set of copy subtasks.

[0010] In an exemplary embodiment, the first execution unit includes:

[0011] A first execution module is used to write task information of the copy subtasks in the group of copy subtasks into the second database of the first storage cluster, so that a task execution node among the multiple task execution nodes searches for the copy subtasks from the second database for execution, thereby completing the distribution of the group of copy subtasks, wherein the first database and the second database are the same database or different databases; wherein the copy subtasks in the group of copy subtasks are designated by the corresponding task execution node by the current node, or are selected for execution in sequence according to the order in which they are searched by the task execution nodes among the multiple task execution nodes.

[0012] In an exemplary embodiment, the device further includes: a second execution unit, configured to update the task status of the first copy task to a distribution completion status if the task information of all copy subtasks in the group of copy subtasks has been written into the second database, wherein the distribution completion status is a status in which the task information of the copy subtasks into which the corresponding copy task is split has been written into the second database; a third execution unit, configured to periodically retrieve the task status of the copy subtasks in the group of copy subtasks; a fourth execution unit, configured to determine, if there is a first copy subtask in the group of copy subtasks with a task status of being executed, based on the execution node information of the first copy subtask, the task execution node corresponding to the first copy subtask, wherein the execution node information is used to indicate the task execution node that executes the corresponding copy subtask; if the task execution node corresponding to the first copy subtask is not in a surviving state, clear the execution node information of the first copy subtask and update the task status of the first copy subtask to an unexecuted state; and a fifth execution unit, configured to update the task status of the first copy task to a completed state if there is no copy subtask in the group of copy subtasks with a task status of either the unexecuted state or the executing state.

[0013] In an exemplary embodiment, the first splitting unit further includes: a second execution module for splitting the data objects indicated by the object list corresponding to the first copy task into a group of K data objects when the task type of the first copy task is a batch copy task in which the data objects to be copied are indicated by an object list, and generating corresponding copy subtasks for the split data object groups to obtain the group of copy subtasks, wherein K is a positive integer greater than or equal to 2; a third execution module for batching the data objects into M data objects as a batch when the task type of the first copy task is a whole-bucket copy task in which the data objects in the specified bucket are copied. Enumerate the data objects in the specified bucket corresponding to the first copy task, and generate corresponding copy subtasks for the data objects listed in different batches, to obtain the set of copy subtasks, wherein M is a positive integer greater than or equal to 2; a fourth execution module is used to enumerate the data objects with the specified prefix corresponding to the first copy task in batches with N data objects as a batch, when the task type of the first copy task is a specified prefix copy task of a copy task for copying data objects with a specified prefix, and generate corresponding copy subtasks for the data objects listed in different batches, to obtain the set of copy subtasks, wherein N is a positive integer greater than or equal to 2.

[0014] In an exemplary embodiment, the apparatus further includes: a second acquisition unit, configured to, when the current node is a task execution node of the first storage cluster, acquire, distributed by the task control node of the first storage cluster and into which the second copy task is split, the second copy task being used to copy a second set of data objects in the first storage cluster to a third storage cluster, and the second copy subtask being used to copy a portion of second data objects in the second set of data objects to the third storage cluster; a first packaging unit, configured to, when there are multiple first-class data objects with a data volume less than or equal to a first data volume threshold in the second data objects copied by the second copy subtask, package at least a portion of the multiple first-class data objects, and copy the packaged data packets to the third storage cluster; and a second splitting unit, configured to, when there are second-class data objects with a data volume greater than a second data volume threshold in the second data objects copied by the second copy subtask and the second-class data objects do not exist in the third storage cluster, split the second-class data objects into multiple sub-data objects, and copy the multiple sub-data objects respectively to the third storage cluster.

[0015] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps in any one of the above method embodiments when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0017] The present application also provides a computer program product, including a computer program, which implements the steps in any of the above method embodiments when the computer program is executed by a processor.

[0018] Through the present application, since the first copy task to be executed can be obtained when the current node is the task control node of the first storage cluster, the first copy task can be used to copy the first data object set of the first storage cluster to the second storage cluster; based on the task type of the first copy task, the first copy task can be split into a group of copy subtasks, wherein the task type of the first copy task can be used to locate the first data object set, and the first copy subtask in a group of copy subtasks can be used to copy part of the first data objects in the first data object set to the second storage cluster, which can improve the accuracy of object copying and avoid unnecessary waste of resources; by distributing a group of copy subtasks to multiple task execution nodes of the first storage cluster, the task execution nodes among the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the distributed copy subtasks, thereby realizing parallel processing of object copy, further improving the efficiency of object copy across storage clusters, and solving the problem of low object copy efficiency in the object copy method across storage clusters in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 This is a schematic diagram of an application scenario of a method for copying objects across storage clusters provided according to an embodiment of the present application.

[0021] Figure 2 This is a flowchart of an optional method for copying objects across storage clusters according to an embodiment of the present application.

[0022] Figure 3This is a system framework diagram of an optional cross-storage cluster object copy according to an embodiment of the present application.

[0023] Figure 4 This is a structural block diagram of an optional cross-storage cluster object copying device according to an embodiment of the present application.

[0024] Figure 5 This is a block diagram of a computer system structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0027] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0028] According to one aspect of an embodiment of the present application, a method for copying an object across a storage cluster is provided. Optionally, in an embodiment of the present application, the above-mentioned method for copying an object across a storage cluster can be applied to, but is not limited to, Figure 1 The hardware environment shown includes a terminal device 102 and a server 104. The server 104 can be connected to the terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) for the terminal device 102 or a client installed on the terminal device 102. A database can be set on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0029] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, or a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: wireless fidelity (Wi-Fi) and Bluetooth. Terminal device 102 may be, but is not limited to, a personal computer (PC), a mobile phone, a tablet computer, etc. Server 104 may be, but is not limited to, a cloud server, a server cluster, or other server types.

[0030] The object copy method across storage clusters of the embodiment of the present application can be executed by the server 104, or by the terminal device 102, or by both the server 104 and the terminal device 102. The object copy method across storage clusters of the embodiment of the present application can be executed by the terminal device 102 or by a client installed thereon.

[0031] Taking the server 104 as an example to execute the object copy method across storage clusters in this embodiment, Figure 2 This is a flow chart of an optional cross-storage cluster object copy method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps S202 to S206.

[0032] Step S202 : when the current node is a task control node of the first storage cluster, obtain a first copy task to be executed, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster.

[0033] Step S204: Based on the task type of the first copy task, split the first copy task into a group of copy subtasks, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy part of the first data objects in the first data object set to the second storage cluster.

[0034] Step S206: distribute a set of copy subtasks to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data object to the second storage cluster according to the copy subtasks in the distributed set of copy subtasks.

[0035] The object copy method across storage clusters in the embodiment of the present application can be applied to the field of data storage technology and to the scenario of copying data objects. In a distributed storage object system, the stored objects are composed of two parts, namely metadata and data content. Metadata refers to users and buckets in the object storage, and data content refers to objects in the bucket. Among them, buckets are containers in the object storage, used to store objects. A site can be an independent object storage system with its own storage gateway and storage pool. A site can have multiple object storage gateway instances, which can independently provide business access to the outside world. The data content and metadata between multiple sites are automatically backed up in an active pull manner through synchronization logs, and ultimately achieve complete consistency of data content and metadata. In addition, the source cluster can refer to the "source" cluster of the data, that is, the storage cluster that stores the original objects, and the target cluster can refer to the "destination" cluster of the data, that is, the storage cluster that receives and saves the copied objects. The source bucket can be the bucket in the source cluster that stores the original objects, and the target bucket can refer to the bucket in the target cluster that is used to receive the copied objects.

[0036] The distributed object storage cluster has storage clusters such as raw material area, processing area, rough processing area, and fine processing area, and there are a large number of data copies between clusters. In the object copy method across storage clusters in the related art, the client (for example, s3cmd) supports copying the object objA in bucket A to the object objB in bucket B. This method can only copy one object at a time, and the source bucket and the target bucket must be in the same cluster. In addition, the client s3cmd will interact with the storage system with five representational state transfer (Restful) requests each time it copies an object. In addition, when the object storage provides a Restful interface, the object objA in bucket A can be directly copied to the object objB in bucket B. This method only requires one interaction between the client and the storage system when copying the object, but it also requires that bucket A and bucket B must be in the same cluster. Therefore, the object copy method across storage clusters in the above-mentioned related art has the problems of low object copy efficiency and many copy interactions.

[0037] Furthermore, the object copying method across storage clusters in the related art also uses the bucket lifecycle for copying. The bucket lifecycle supports copying objects in a bucket to a target bucket in the target storage cluster. This method is applicable to a one-to-one correspondence between source and target buckets, that is, all objects in the source bucket or all objects with a specified prefix are copied to a target bucket, and the object names remain unchanged after the copy. In addition, in order not to affect business, the bucket lifecycle generally operates between 12:00 and 6:00 a.m. every day. This method cannot meet the user's instant copy scenario. If the user only copies some objects with different prefixes in the bucket to different target buckets, this method cannot meet customer needs.

[0038] Furthermore, related cross-storage cluster object copying methods also employ multi-site data synchronization. Multi-site data synchronization allows for the complete replication of all objects in a bucket at one site to a target cluster. This assumes that the source and target clusters already form a multi-site environment. In this case, data uploaded from the source site is asynchronously replicated to the bucket of the same name at the target site. This approach is suitable for scenarios where a synchronous replication relationship has already been established between clusters and all data on the source cluster needs to be replicated to the target cluster. Clearly, multi-site data synchronization cannot meet customer needs.

[0039] In order to at least partially solve the above problems, in this embodiment, when the current node is the task control node of the first storage cluster, the first copy task to be executed can be obtained, and the first copy task can be used to copy the first data object set of the first storage cluster to the second storage cluster; based on the task type of the first copy task, the first copy task can be split into a group of first copy subtasks, wherein the task type of the first copy task can be used to locate the first data object set, and the copy subtasks in a group of copy subtasks can be used to copy part of the first data objects in the first data object set to the second storage cluster, which can improve the accuracy of object copying and avoid unnecessary waste of resources; by distributing a group of copy subtasks to multiple task execution nodes of the first storage cluster, the task execution nodes in the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks in the distributed group of copy subtasks, thereby realizing parallel processing of object copying and further improving the efficiency of object copying across storage clusters.

[0040] In the embodiment of the present application, the first storage cluster may be a source storage cluster, and the second storage cluster may be a target storage cluster, without specific limitation herein.

[0041] When the current node is the task control node of the first storage cluster, a first copy task to be executed is obtained. The first storage cluster may be provided with multiple nodes. When the current node is the task control node, the first copy task to be executed is obtained. The first copy task is used to copy the first data object set in the first storage cluster to the second storage cluster. It is understood that after obtaining the first copy task, the first copy task can copy the first data object set in the first storage cluster to the second storage cluster, so that the first data object set is also in the second storage cluster.

[0042] It should be noted that the first data object set of the first storage cluster can be stored in the storage layer, the first data object can be copied to the storage layer in the second storage cluster, the first copy task to be executed can be stored in the database, and the first copy task to be executed can be obtained from the database.

[0043] In an optional embodiment, when the current node is a task control node of the first storage cluster, the first copy task to be executed is obtained. The first storage cluster may include multiple nodes, one of which may be a task control node (or called a master monitoring node). The task control node is used to control the copy task, for example, to control the distribution of the copy task, monitor the execution process of the copy task, etc. The remaining nodes may be task execution nodes, which are nodes that execute the copy task or the copy subtask. The task control node may be designated, or may be elected by the node, or may be determined in other ways, which is not limited in this embodiment.

[0044] Two kinds of threads can be run on the current node, i.e., a control thread and an execution thread, wherein the control thread can be used to periodically detect whether the node is a task control node. If it is a task control node, it can obtain the copy task to be executed from the specified storage area (e.g., a specified database). The copy task to be executed here can be a copy task that has not been executed, or a copy task that has been partially executed. The current node can also be a task execution node at the same time, and the execution thread on it can obtain the copy task or the copy subtasks that the copy task is split into, and perform cross-storage cluster copying of data objects. It is possible that the same node can be only one of the task control node and the task execution node at the same time. In this case, when it is detected that the current node is switched from a task execution node to a task control node, when there is a copy subtask that has not been executed by the current node, the current node can mark the unexecuted copy subtask as the state of the task execution node to be switched, and mark the copied data object to avoid duplicate copying of data.

[0045] The first copy task obtained is split into a set of copy subtasks based on the task type of the first copy task. The task type of the first copy task is used to locate the first set of data objects. It is understood that, based on the task type of the first copy task, the first copy task is split into multiple copy subtasks to form the first copy task.

[0046] The task type of the first copy task is used to locate the first data object set. That is, the task type of the first copy task can be used to locate the number of the first data object set, the size of the first data object set, and the copy method of the data objects in the first data object set. For example, the task type of the first copy task can be used to copy a single object, a batch of objects, an entire bucket of objects, or all objects in the bucket with a specified prefix.

[0047] For example, if the task type of the first copy task is to copy the entire bucket, the objects in the bucket can be divided into multiple first copy subtasks evenly or according to a preset number based on the number and size of objects in the bucket. If the task type of the first copy task is to copy based on a specified prefix, the tasks can be divided into multiple first copy subtasks based on the different specified prefixes or the number of objects. If the task type of the first copy task is to batch copy objects in a list, the tasks can be divided into multiple copy subtasks based on the preset number of objects in the list.

[0048] The copy subtasks in the group of copy subtasks are used to copy a portion of the first data objects in the first data object set to the second storage cluster. That is, the group of copy subtasks includes multiple copy subtasks, each copy subtask copies a portion of the first data objects in the first data object set to the second storage cluster, and the multiple copy subtasks copy all the first data objects in the first data object set to the second storage cluster.

[0049] A set of copy subtasks is distributed to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks of the distributed set of copy subtasks. It can be understood that a set of copy subtasks is distributed to multiple task execution nodes in the first storage cluster, and after each task execution node receives the copy subtask, it starts to execute the data object copy operation, thereby copying some of the first data objects in the first data object set to the second storage cluster. Multiple task execution nodes execute the data object copy operation to copy all the first data objects in the first data object set to the second storage cluster. Each task execution node can execute one copy subtask or multiple copy subtasks. For example, one task execution node executes three copy subtasks.

[0050] For example, a node in the first storage cluster can be selected to act as a task control node, obtain the pending first copy task, split it into several pending copy subtasks, and mark the task as completed after all the copy subtasks are completed. All nodes in the first storage cluster can act as task execution nodes, obtain pending copy subtasks, and copy the objects in the copy subtasks across clusters to the target storage cluster (the second storage cluster).

[0051] The above-mentioned copying of the corresponding first data object to the second storage cluster may include: the task execution node executes the object copy operation, first obtains a copy subtask to be executed, and then copies the objects to the target storage cluster (the second storage cluster) in sequence according to the object. After the second storage cluster receives the task, it parses and obtains the original object, performs permission checks, and writes to the storage layer.

[0052] The task execution nodes among the plurality of task execution nodes execute the copy subtasks in the distributed set of copy subtasks. That is, the plurality of task execution nodes may simultaneously copy the corresponding first data objects to the second storage cluster according to the distributed copy subtasks, or may copy the corresponding first data objects to the second storage cluster according to the distributed copy subtasks in the order in which the copy subtasks were generated or written.

[0053] In another optional embodiment, when obtaining the first copy task to be executed, the first copy task can be sorted according to the urgency or business priority of the first copy task. When splitting and distributing a group of copy subtasks, copy subtasks with high priority can be processed first, and their copy subtasks are preferentially allocated resources in the first storage cluster.

[0054] Through the embodiments of the present application, when the current node is a task control node of the first storage cluster, the first copy task to be executed can be obtained, and the first copy task can be used to copy the first data object set of the first storage cluster to the second storage cluster; based on the task type of the first copy task, the first copy task can be split into a group of copy subtasks, wherein the task type of the first copy task can be used to locate the first data object set, and the copy subtasks in a group of copy subtasks can be used to copy part of the first data objects in the first data object set to the second storage cluster, which can improve the accuracy of object copying and avoid unnecessary waste of resources; by distributing a group of copy subtasks to multiple task execution nodes of the first storage cluster, the task execution nodes in the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks of the distributed group of copy subtasks, thereby realizing parallel processing of data object copy, further improving the efficiency of object copy across storage clusters, and solving the problem of low data object copy efficiency in the object copy method across storage clusters in the related art.

[0055] In some exemplary embodiments, the task information of the copy task issued by the service platform to the first storage cluster is stored in the first database of the first storage cluster.

[0056] When the current node is the task control node of the first storage cluster, obtaining the first copy task to be executed includes: when the current node is the task control node of the first storage cluster, searching the first database for copy tasks whose task status belongs to a specified state set, wherein the specified state set includes an unexecuted state and an incomplete split state, and the incomplete split state is a state in which the corresponding copy task has begun to be split into copy subtasks, but the splitting has not been completed.

[0057] When searching for a copy task whose task status belongs to the specified state set, the copy task whose task status belongs to the specified state set and whose creation time is the earliest is determined as the first copy task to be executed. Among them, when the task status of the first copy task is the unexecuted state, the task status of the first copy task is modified to the split unfinished state.

[0058] The task information of the copy task issued by the service platform to the first storage cluster is stored in the first database of the first storage cluster. That is, the task information of the copy task issued by the service platform to the first storage cluster via an interface (e.g., a RESTful interface) is stored in the first database of the first storage cluster. The service platform can be an application platform, a cloud service platform, or a data management platform, etc., and is not specifically limited here.

[0059] For example, the business platform can use a Restful interface to send a quick copy task to the object storage (the first storage cluster). The first storage cluster performs checks such as parameter entry and task resending. If the checks fail, it directly reports an error to the business platform. If the checks pass, the first storage cluster can persist the task information to the first database, record the task status as pending, and return a success message to the business platform.

[0060] For example, the business platform can use a private interface to send different types of copy tasks to the source storage cluster (first storage cluster), such as batch copy tasks, whole bucket copy tasks, and copy tasks with specified prefixes, while specifying the service access point of the target cluster (second storage cluster).

[0061] In this embodiment, the task information of the copy task issued by the business platform to the first storage cluster is stored in the first database of the first storage cluster. It is understandable that any node in the storage cluster may receive the copy task issued by the business platform and store the task information of the copy task in the database.

[0062] When the current node is the task control node of the first storage cluster, a search is performed from the first database for copy tasks whose task status belongs to a specified state set. The specified state set includes an unexecuted state and an unsplit state. The unsplit state is a state where the corresponding copy task has begun to be split into copy subtasks but the splitting has not been completed. The unexecuted state can be a state where the task is being created and is waiting to be processed.

[0063] When a copy task whose task status belongs to a specified state set is searched, the copy task whose task status belongs to the specified state set and whose creation time is the earliest is determined as the first copy task to be executed. It is understandable that when a copy task whose task status belongs to an unexecuted state is searched, the copy task whose task status belongs to an unexecuted state and whose creation time is the earliest is determined as the first copy task to be executed; or when a copy task whose task status belongs to an unsplit state is searched, the copy task whose task status belongs to an unsplit state and whose creation time is the earliest is determined as the first copy task to be executed.

[0064] Wherein, when the task status of the first copy task is the unexecuted state, the task status of the first copy task is modified to the split unfinished state. In addition, when the task status of the first copy task is the split unfinished state, the task status of the first copy task is not modified.

[0065] For example, tasks that are being split and not completed and tasks that have not been executed are retrieved from the first database, and the task with the earliest creation time is selected for execution; if the task is in the unexecuted state, the task status is changed to the unfinished split state, and the time for starting the splitting of the task is changed to the current time; if the task is in the unfinished split state, the task status is not changed.

[0066] It should be noted that the task information of the copy task may include but is not limited to the task identifier (each copy task has a unique identifier (ID) to facilitate user query, cancellation, and recovery operations), the first storage cluster information (that is, the source storage cluster information, including the source storage cluster access point, bucket name, object key, and authentication information (Access Key and Secret). Key), second storage cluster information (i.e., target storage cluster information, for example, the access point of the target storage cluster, the target bucket name, the target object key, etc., used for data writing operations), task type (indicating the type of copy task, such as batch copy, whole bucket copy, or specified prefix copy, etc.), task status information (the current status of the copy task, such as "not executed", "in execution", "completed", "canceled", or "failed", where the "in execution" status indicates that the task has started to split into subtasks but has not been completed), task creation time (the specific timestamp when the copy task was created), task tag (an identifier used to indicate the current subtask split position in the data object set), and the total number of expected processed objects (the number of data objects planned to be processed in the task). These can be set according to actual conditions and are not specifically limited in the embodiments of the present application.

[0067] Through the embodiment of the present application, the business platform sends the task information of the copy task to the first storage cluster. The copy task is stored in the first database. When the task control node searches for a task to be executed, it can search the first database for a copy task whose task status belongs to a specified state set. This facilitates the management and monitoring of the task status. Storing the task information of the copy task in the first database helps the system's fault tolerance and recovery. Even after a failure occurs, the system can restore the task status based on the stored information and continue the unfinished task.

[0068] In some exemplary embodiments, based on the task type of the first copy task, the first copy task is split into a group of copy subtasks, including: when the task status of the first copy task is an incomplete split state, based on the task tag of the first copy task, the current split position is determined, wherein the task tag of the first copy task is used to indicate the object position of the first first data object in the first data object set that has not been split into the copy subtask.

[0069] Based on the task type of the first copy task, the first data object located after the current split position in the first data object set is split into copy subtasks to obtain a group of copy subtasks.

[0070] When the task status of the first copy task is in an incomplete split state, the current split position is determined based on the task marker of the first copy task. In this embodiment, when the task status of the first copy task is in an incomplete split state, the split position of the first copy task is determined based on the task marker of the first copy task. The task marker can be an identifier stored in the task information for indicating the current progress of the copy task. For example, the task marker can be a pointer that points to the position of the first data object in the first data object set that has not yet been included in the subtask processing.

[0071] For example, when it is detected that the first copy task is in an uncompleted split state, it can be determined where the current split operation should start based on a pre-recorded task marker.

[0072] The task tag of the first copy task is used to indicate the object position of the first first data object in the first data object set that has not been split into a copy subtask. In other words, the task tag of the first copy task can be used to indicate the object position of the last first data object in the first data object set that has been split into a copy subtask.

[0073] Based on the task type of the first copy task, the first data object in the first data object set that is located after the current split position is split into copy subtasks to obtain a set of first copy subtasks. For example, based on the task type of the first copy task, the unprocessed portion of the first data object set, that is, the first data object after the current split position, is split into a set of subtasks.

[0074] Through the embodiment of the present application, based on the task mark of the first copy task, the current split position is determined, and based on the task type of the first copy task, the first data object in the first data object set that is located after the current split position is split into copy subtasks, thereby obtaining a group of copy subtasks, avoiding repeated splitting, improving the efficiency of data object copying, and reducing resource waste.

[0075] In some exemplary embodiments, a group of copy subtasks is distributed to multiple task execution nodes of a first storage cluster, including: writing task information of the copy subtasks in the group of copy subtasks into a second database of the first storage cluster, so that the task execution nodes among the multiple task execution nodes search for the copy subtasks from the second database for execution, thereby completing the distribution of the group of copy subtasks, wherein the first database and the second database are the same database or different databases.

[0076] The copy subtasks in a group of copy subtasks are assigned corresponding task execution nodes by the current node, or are selected for execution in sequence according to the order in which they are searched by the task execution nodes in the plurality of task execution nodes.

[0077] In this embodiment, the task information of the copy subtasks in a group of copy subtasks is written into the second database of the first storage cluster, so that the task execution node in the plurality of task execution nodes searches for the copy subtask from the second database for execution. The task information of the above-mentioned copy subtasks may include but is not limited to the copy subtask ID (a unique identifier for each copy subtask), the copy task ID to which it belongs (indicating the main copy task to which the sub-copy task belongs), the source object list (specifically listing the data objects to be copied and their locations in the source storage cluster (the first storage cluster)), the target object location (specifying the location information of each source object copied to the target storage cluster (the second storage cluster)), the copy subtask status information (the execution status of the copy subtask, such as "not executed", "in execution", "completed" or "canceled"), the task execution node information (recording the node ID that will execute the copy subtask for task scheduling and monitoring), the write time (the time when the copy subtask information is written to the second database, used to affect the selection order of the execution node), the copy success / failure object count (for each copy subtask, recording the number of data objects that are successfully and failed to be copied), which can be set according to actual conditions and is not specifically limited in the embodiment of the present application.

[0078] The task information of a copy subtask in a group of copy subtasks is written into the second database of the first storage cluster, so that a task execution node among multiple task execution nodes searches for the first copy subtask from the second database, so that the copy subtask can be selected and executed in sequence according to the corresponding task execution node specified by the current node or according to the order in which the task execution nodes among the multiple task execution nodes are searched.

[0079] The copy subtasks in a group of copy subtasks are assigned corresponding task execution nodes by the current node, or are selected for execution in the order in which they are searched by the task execution nodes in the plurality of task execution nodes. The current node may be a task control node of the first storage cluster.

[0080] For example, the copy subtasks of a group of copy subtasks can be directly assigned to a task execution node by the current node, or, if not specified, the task execution node selects to execute according to the writing order of the copy subtasks in the second database. This makes the tasks more efficiently allocated and provides flexibility in task scheduling.

[0081] Through the embodiment of the present application, the subtask information of the copy subtasks in a group of copy subtasks is stored in the second database, and can be searched and executed in parallel by multiple task execution nodes, effectively improving the parallelism of data object copying, thereby further improving the efficiency of data object copying.

[0082] In some exemplary embodiments, after writing the task information of the copy subtasks in the set of copy subtasks into the second database, the method further includes:

[0083] 1. When the task information of all copy subtasks in a group of copy subtasks has been written into the second database, the task status of the first copy task is updated to the distribution completion status, wherein the distribution completion status is the status in which the task information of the copy subtasks into which the corresponding copy task is split has been written into the second database.

[0084] 2. Periodically retrieve the task status of a copy subtask in a group of copy subtasks.

[0085] 3. In a group of copy subtasks, if there is a first copy subtask whose task status is in the executing state, determine the task execution node corresponding to the first copy subtask based on the execution node information of the first copy subtask, wherein the execution node information is used to indicate the task execution node that executes the corresponding copy subtask; if the task execution node corresponding to the first copy subtask is not in the surviving state, clear the execution node information of the first copy subtask, and update the task status of the first copy subtask to the unexecuted state.

[0086] If there is no copy subtask in a group of copy subtasks whose task status is either the unexecuted state or the executing state, the task status of the first copy task is updated to the completed state.

[0087] When the task information of all copy subtasks in a group of copy subtasks has been written into the second database, the task status of the first copy task is updated to a distribution completed state, wherein the distribution completed state can be a state in which the task information of the copy subtasks into which the corresponding copy task is split has been written into the second database. In an optional embodiment, the distribution completed state can also be a state in which the copy subtasks into which the corresponding copy task is split have been distributed to each task execution node.

[0088] For example, a group of copy subtasks are distributed to multiple task execution nodes of the first storage cluster. If the object distribution of the current task is completed, the task status is modified to distribution completed, and then the next task is processed.

[0089] Periodically retrieve the task status of the copy subtasks in the group of copy subtasks. For example, the task control node can periodically check the status of a group of copy subtasks in the second database, and learn the task status and progress of each copy subtask through query.

[0090] In a group of copy subtasks, if there is a first copy subtask with a task status of being executed, the task execution node corresponding to the first copy subtask is determined based on the execution node information of the first copy subtask, wherein the execution node information is used to indicate the task execution node that executes the corresponding copy subtask. It should be noted that the execution node information may include but is not limited to: node identification (each execution node has a unique identifier used to locate the node within the cluster, and the identifier may be the name, number, etc. of the node), node status (current operating status information of the execution node, such as "online", "offline", "busy" or "idle"), and task execution progress (execution progress of the subtask currently being processed, such as the number of completed data objects, total data transmission volume, execution time, etc.).

[0091] When the task execution node corresponding to the first copy subtask is not in a surviving state, the execution node information of the first copy subtask is cleared, and the task status of the copy subtask is updated to an unexecuted state. That is, when the task execution node corresponding to the first copy subtask is in a non-surviving (dead) state, the execution node information of the first copy subtask is cleared, and the task status of the first copy subtask is updated to an unexecuted state, so that the first copy subtask can be executed by other task execution nodes.

[0092] For example, the task execution node can periodically send heartbeat information (survival information) to the database to prove that the current execution thread (task execution node) is alive.

[0093] If there is no copy subtask in a group of copy subtasks whose task status is either the unexecuted state or the executing state, the task status of the first copy task is updated to the completed state.

[0094] For example, when all copy subtasks have been successfully executed and no copy subtask is in the executing or unexecuted state, the state of the first copy task is updated to the completed state.

[0095] For example, when a subtask encounters an execution node failure, the system automatically reassigns the subtask and updates its status after successful re-execution. If, at a certain point, all subtasks have completed or have been marked as failed and are no longer reassigned (for example, all failed retry opportunities have been exhausted), the first copy task status is updated to "Completed." If, during the copy process, a task execution node suddenly goes offline, causing the load copy subtask to fail, but these copy subtasks are reassigned to other online nodes, and all subtasks are eventually completed, the first copy task will be marked as "Completed."

[0096] In this embodiment, the tasks that have been distributed are retrieved regularly, and for each task, it is retrieved whether there is an unexecuted copy subtask. If not, the task status is modified to completed.

[0097] Through the embodiment of the present application, when the task information of all copy subtasks of the first copy task has been written to the second database, the task status is updated to distribution completion, and the task status is updated to completed status after all copy subtasks are completed, thereby improving the efficiency and accuracy of task management. When it is detected that the task execution node is not in a surviving state, the copy subtask status automatically rolls back to the unexecuted state, and the copy subtask can be reallocated to other surviving nodes, thereby improving the continuity and flexibility of task execution and improving the overall execution efficiency and reliability. Once all subtasks have been executed, the status of the first copy task is updated to completed, which is convenient for the system and the user to confirm the final status of the task.

[0098] In some exemplary embodiments, based on the task type of the first copy task, the first copy task is split into a group of copy subtasks, including:

[0099] 1. When the task type of the first copy task is a batch copy task in which the data objects to be copied are indicated by an object list, the data objects indicated by the object list corresponding to the first copy task are split into groups of K data objects, and corresponding copy subtasks are generated for the split data object groups to obtain a group of copy subtasks, where K is a positive integer greater than or equal to 2.

[0100] 2. When the task type of the first copy task is a whole-bucket copy task that copies data objects in a specified bucket, the data objects in the specified bucket corresponding to the first copy task are listed in batches of M data objects, and corresponding copy subtasks are generated for the data objects listed in different batches to obtain a group of copy subtasks, where M is a positive integer greater than or equal to 2.

[0101] 3. In the case where the task type of the first copy task is a specified prefix copy task that copies data objects with a specified prefix, the data objects with the specified prefix corresponding to the first copy task are listed in batches with N data objects as a batch, and corresponding copy subtasks are generated for the data objects listed in different batches to obtain a group of copy subtasks, where N is a positive integer greater than or equal to 2.

[0102] In the case where the task type of the first copy task is a batch copy task in which the data objects to be copied are indicated by an object list, the task is split into groups of K data objects, where K can be an arbitrarily set positive integer greater than or equal to 2. It should be noted that after the task is split into multiple groups based on K data objects as a group, the remaining Y data objects can form a group, where Y is a positive integer less than K.

[0103] For example, the task type of the first copy task is object list, and the number of objects in the object list is 10090. Then a copy subtask is generated for every 100 objects in the object list. After splitting into 100 groups of copy subtasks, there are 90 data objects left. These 90 objects can form a group of copy subtasks.

[0104] If the first copy task is a full-bucket copy task that copies data objects in a specified bucket, M (M is a positive integer greater than or equal to 2) data objects are listed from the bucket batch by batch, generating one copy subtask for each batch. Note that, based on M data objects as a batch, after listing the objects in the bucket, the remaining X data objects can be grouped together, where X is a positive integer less than M.

[0105] For example, if the first copy task is a bucket-wide copy task, 1000 objects in the bucket are enumerated in an unordered manner based on the marker to generate a first copy subtask. If the initial status of the copy subtask is not executed, the generated copy subtasks are batch-written to the first database or the second database, and the marker in the task is updated simultaneously.

[0106] In the case where the task type of the first copy task is a batch copy task in which the data objects to be copied are indicated by an object list, data objects with a specified prefix are listed in batches, with N (N is a positive integer greater than or equal to 2) data objects as a batch. Similar to the case of the above-mentioned whole bucket copy task, each batch of data objects will be converted into a copy subtask, and finally combined into a group of copy subtasks. It should be noted that, based on N data objects as a batch, after the data objects with a specified prefix in the bucket are listed in sequence, the remaining Z data objects with the specified prefix can be grouped together, where Z is a positive integer less than N.

[0107] For example, if the first copy task is a bucket-wide copy task, a copy subtask is generated by enumerating 10,000 objects from the bucket using a marker in an unordered enumeration manner. If the initial status of the first copy subtask is not executed, the generated copy subtasks are batch-written to the first database or the second database, and the marker in the task is updated simultaneously.

[0108] By splitting the first copy task into multiple first copy subtasks, the present invention can fully utilize the advantages of multi-node parallel processing and improve the efficiency of data object copying. At the same time, according to different task types (batch, whole bucket, or specified prefix copy), task processing is made more flexible and diverse.

[0109] In some exemplary embodiments, the above method further includes:

[0110] 1. When the current node is the task execution node of the first storage cluster, obtain the second copy subtask distributed by the task control node of the first storage cluster and split into the second copy task, wherein the second copy task is used to copy the second data object set in the first storage cluster to the third storage cluster, and the second copy subtask is used to copy part of the second data objects in the second data object set to the third storage cluster.

[0111] 2. When there are multiple first-category data objects whose data volume is less than or equal to the first data volume threshold in the second data object copied by the second copy subtask, at least part of the multiple first-category data objects are packaged, and the packaged data packets are copied to the third storage cluster.

[0112] 3. When there is a second-category data object with a data volume greater than a second data volume threshold in the second data object copied by the second copy subtask, and there is no second-category data object in the third storage cluster, the second-category data object is split into multiple sub-data objects, and the multiple sub-data objects are copied to the third storage cluster respectively.

[0113] It should be noted that the third storage cluster may be the same cluster as the second storage cluster, or may be a different cluster from the second storage cluster.

[0114] In this embodiment, the second copy task is used to copy the second data object set in the first storage cluster to the third storage cluster. The second copy task can be split into multiple second copy subtasks, and the second copy subtasks are used to copy part of the second data objects in the second data object set to the third storage cluster.

[0115] When there are multiple first-category data objects whose data volume is less than or equal to the first data volume threshold in the second data object copied by the second copy subtask, at least part of the multiple first-category data objects are packaged, and the packaged data packets are copied to the third storage cluster.

[0116] For example, the first type of data objects are small objects (for example, objects with smaller memory capacity). Small objects can be executed in batches, merged and packaged. Multiple small objects are packaged together and delivered to the destination (third storage cluster) for storage at one time. After the destination storage receives the task, it parses the original object, performs permission checks, and writes it to the storage layer.

[0117] If there is a second-category data object with a data volume greater than a second data volume threshold in the second data object copied by the second copy subtask, and there is no second-category data object in the third storage cluster, the second-category data object is split into multiple sub-data objects, and the multiple sub-data objects are copied to the third storage cluster respectively.

[0118] For example, if the second type of data object is a large object (an object exceeding a certain size, such as an object with a large memory size), we first try to check whether an object with the same name exists in the destination cluster (the third storage cluster) and whether the data is consistent. If so, the object transfer is complete; otherwise, we copy the object using the multi-segment upload method.

[0119] In an optional embodiment, the number of copy subtasks in the database of the third storage cluster may be pre-set to limit the number of first copy subtasks in the first database of the third storage cluster. For example, the total number of first copy subtasks in the first database may be limited to 10,000, and creation of first copy subtasks may be stopped if the number exceeds a threshold (i.e., exceeds 10,000).

[0120] Through the embodiments of the present application, the first type of data objects (small files) are packaged and transmitted, which significantly reduces the number of network interactions, thereby accelerating the transmission speed of the small file collection. The second type of data objects (large files) are split and transmitted, which speeds up the transmission rate of the second type of data, effectively shortens the completion time of the entire copy operation, and improves the copy efficiency of data objects.

[0121] In some exemplary embodiments, the above method further includes: modifying the task status of a first copy subtask of a group of first copy subtasks to a canceled state.

[0122] For example, when the object service (first storage cluster) receives a request to cancel a copy task, it first obtains the task information from the database, performs a status check, and modifies the task status in the database to canceled after the check passes. It then sends a message to notify the task control node gateway, and directly returns to the business platform after the notification is successful. If the check fails, it directly returns to the business platform that the operation failed. After receiving the notification request, the gateway puts the task to be canceled into the memory, and then requests a return; the task control thread (task control node) regularly checks the memory to see if there is a task to be canceled. If so, it first determines whether the task to be canceled is being split. If so, it will no longer be split into subtasks; after the task execution thread obtains a subtask, it determines the execution status of the task in which the subtask is located. If the task has been canceled, the subtask is skipped.

[0123] In some exemplary embodiments, the above method further includes: restoring the first copy task whose task status is canceled.

[0124] For example, after receiving a request to restore a task, the object service (the first storage cluster) retrieves the task information from the database. If the task is not in the canceled state, an error is directly returned to the business platform. If it is in the canceled state and the task start time in the task information is not empty, the task status is changed to "in progress." If the task start time in the task information is empty, the task status is changed to "not executed." The task recovery result is returned to the business platform. The task control thread (task control node) can sequentially retrieve tasks and split them into subtasks based on the task status and creation time. The execution thread will sequentially retrieve subtasks for processing based on the subtask status and creation time.

[0125] The object copy method across storage clusters in the embodiment of the present application is explained below with reference to optional examples. Figure 3 This is a system framework diagram of an optional cross-storage cluster object copy according to an embodiment of the present application, such as Figure 3 As shown, an exemplary description is given of the task distribution of the cross-storage cluster object copy method in the distributed object storage cluster in the raw material area.

[0126] The control thread periodically checks whether its node is the primary monitoring (mon) node. If so, it executes subsequent processes; otherwise, it waits for the next execution cycle to recheck. If the control thread is the primary monitoring node, it retrieves tasks from the Elasticsearch (ES) database, a distributed search and analysis tool. The control thread searches for both currently executing and unexecuted tasks, selecting the task with the earliest creation time for execution. If the task is unexecuted, the task status is changed to "in progress" and the task's start execution time is changed to the current time. If the task is in progress, the task status remains unchanged. The execution task is split into subtasks, obtaining a data object collection from the storage layer to enumerate objects and generate subtasks. The control thread writes subtasks in batches to ES and updates task information. The control thread includes a monitoring process that monitors ES for task completion and execution node survival. The execution thread (task execution node) processes the subtasks and can periodically send heartbeat messages (survival information) to the database (ES) to verify the current execution thread's survival. The execution thread retrieves the oldest unexecuted subtask and the status of the task containing it. If it is in progress, execution continues; if it is canceled, execution of the subtask is skipped; if it fails, the subtask is deleted. The execution thread changes the subtask status to in progress, sets the start time to the current time, and updates the subtask with the current node information. If a conflict occurs when modifying the subtask status, the subtask is skipped and the next subtask is executed. The execution thread obtains the object list from the subtask and performs a data copy. After the copy is complete, the execution results of the object list are updated to the database in batches, the subtasks are deleted from the database, and the task information is updated with the number of successful and failed executions. If the subtask is a full-bucket or prefix-specific subtask and verification fails due to an incorrect authentication key, the task execution status is changed to failed, and the number of successful and failed executions in the task information is updated. After obtaining the subtask, the execution thread copies the objects in the subtask to the target storage cluster based on the objects in the subtask. Different data transfer methods are used for objects of different sizes.

[0127] The task splitting mechanism in the embodiment of the present application can realize multi-concurrency, high-performance data copying. The merging and sending of small objects and the concurrent fragmented transmission of large objects greatly improve the data transmission performance. In addition, it has excellent node fault tolerance and can meet the user's various cross-cluster copy needs. Compared with traditional copy solutions, it is more flexible and efficient, enriches the functional diversity of distributed object storage, and brings a better user experience.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0129] Embodiments of the present application also provide a cross-storage cluster object copying device, which can be used to implement a cross-storage cluster object copying method provided in any of the above embodiments. Details already described are not repeated here. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0130] Figure 4 This is a structural block diagram of an optional cross-storage cluster object copy device according to an embodiment of the present application, such as Figure 4 As shown in , the object copy device across storage clusters includes a first obtaining unit 402 , a first splitting unit 404 and a first executing unit 406 .

[0131] The first acquiring unit 402 is configured to acquire a first copy task to be executed when the current node is a task control node of the first storage cluster, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster.

[0132] The first splitting unit 404 is used to split the first copy task into a group of first copy subtasks based on the task type of the first copy task, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy part of the first data objects in the first data object set to the second storage cluster.

[0133] The first execution unit 406 is configured to distribute a set of copy subtasks to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data object to the second storage cluster according to the distributed copy subtasks.

[0134] It should be noted that the first acquiring unit 402 in this embodiment can be used to execute the above step S202, the first splitting unit 404 in this embodiment can be used to execute the above step S204, and the first executing unit 406 in this embodiment can be used to execute the above step S206.

[0135] Through the embodiments of the present application, when the current node is a task control node of the first storage cluster, the first copy task to be executed can be obtained, and the first copy task can be used to copy the first data object set of the first storage cluster to the second storage cluster; based on the task type of the first copy task, the first copy task can be split into a group of copy subtasks, wherein the task type of the first copy task can be used to locate the first data object set, and the copy subtasks in a group of copy subtasks can be used to copy part of the first data objects in the first data object set to the second storage cluster, which can improve the accuracy of object copying and avoid unnecessary waste of resources; by distributing a group of copy subtasks to multiple task execution nodes of the first storage cluster, the task execution nodes among the multiple task execution nodes copy the corresponding first data objects to the second storage cluster according to the copy subtasks of the distributed group of copy tasks, thereby realizing parallel processing of object copying and further improving the efficiency of object copying across storage clusters.

[0136] In an exemplary embodiment, task information of the copy task issued by the service platform to the first storage cluster is stored in the first database of the first storage cluster.

[0137] The above-mentioned first acquisition unit 402 includes: a first search module, which is used to search for copy tasks whose task status belongs to a specified state set from the first database when the current node is the task control node of the first storage cluster, wherein the specified state set includes an unexecuted state and an incomplete split state, and the incomplete split state is a state in which the corresponding copy task has begun to be split into copy subtasks but the splitting has not been completed; a first determination module, which is used to, when a copy task whose task status belongs to the specified state set is searched, determine the copy task whose task status belongs to the specified state set and has the earliest creation time as the first copy task to be executed, wherein, when the task status of the first copy task is the unexecuted state, the task status of the first copy task is modified to the incomplete split state.

[0138] In an exemplary embodiment, the above-mentioned first splitting unit 404 includes: a second determination module, which is used to determine the current split position based on the task mark of the first copy task when the task status of the first copy task is the incomplete split state, wherein the task mark of the first copy task is used to indicate the object position of the first first data object in the first data object set that has not been split into the copy subtask; a first splitting module, which is used to split the first data object in the first data object set that is located after the current split position into the copy subtask based on the task type of the first copy task, to obtain the set of copy subtasks.

[0139] In an exemplary embodiment, the above-mentioned first execution unit 406 includes: a first execution module, used to write the task information of the copy subtasks in the group of copy subtasks into the second database of the first storage cluster, so that the task execution node among the multiple task execution nodes searches for the copy subtasks from the second database for execution, thereby completing the distribution of the group of copy subtasks, wherein the first database and the second database are the same database or different databases; wherein the copy subtasks in the group of copy subtasks are designated by the corresponding task execution node by the current node, or are selected for execution in sequence according to the order in which they are searched by the task execution nodes among the multiple task execution nodes.

[0140] In an exemplary embodiment, the device further includes: a second execution unit, configured to update the task status of the first copy task to a distribution completion status if the task information of all copy subtasks in the group of copy subtasks has been written into the second database, wherein the distribution completion status is a status in which the task information of the copy subtasks into which the corresponding copy task is split has been written into the second database; a third execution unit, configured to periodically retrieve the task status of the copy subtasks in the group of copy subtasks; a fourth execution unit, configured to determine, if there is a first copy subtask in the group of copy subtasks with a task status of being executed, based on the execution node information of the first copy subtask, the task execution node corresponding to the first copy subtask, wherein the execution node information is used to indicate the task execution node that executes the corresponding copy subtask; if the task execution node corresponding to the first copy subtask is not in a surviving state, clear the execution node information of the first copy subtask and update the task status of the first copy subtask to an unexecuted state; and a fifth execution unit, configured to update the task status of the first copy task to a completed state if there is no copy subtask in the group of copy subtasks with a task status of either the unexecuted state or the executing state.

[0141] In an exemplary embodiment, the first splitting unit 404 further includes: a second execution module for splitting the data objects indicated by the object list corresponding to the first copy task into a group of K data objects when the task type of the first copy task is a batch copy task in which the data objects to be copied are indicated by an object list, and generating corresponding copy subtasks for the split data object groups to obtain the group of copy subtasks, wherein K is a positive integer greater than or equal to 2; a third execution module for splitting the data objects indicated by the object list corresponding to the first copy task into a group of K data objects when the task type of the first copy task is a batch copy task in which the data objects in the specified bucket are copied, and generating corresponding copy subtasks for the split data object groups to obtain the group of copy subtasks, wherein K is a positive integer greater than or equal to 2; and a third execution module for splitting the data objects indicated by the object list corresponding to the first copy task into a group of M data objects as a batch when the task type of the first copy task is a whole bucket copy task in which the data objects in the specified bucket are copied. The data objects in the specified bucket corresponding to the first copy task are listed in batches, and corresponding copy subtasks are generated for the data objects listed in different batches to obtain the group of copy subtasks, wherein M is a positive integer greater than or equal to 2; the fourth execution module is used to list the data objects with the specified prefix corresponding to the first copy task in batches with N data objects as a batch, when the task type of the first copy task is a specified prefix copy task of a copy task that copies data objects with a specified prefix, and corresponding copy subtasks are generated for the data objects listed in different batches to obtain the group of copy subtasks, wherein N is a positive integer greater than or equal to 2.

[0142] In an exemplary embodiment, the apparatus further includes: a second acquisition unit, configured to, when the current node is a task execution node of the first storage cluster, acquire, distributed by the task control node of the first storage cluster and into which the second copy task is split, the second copy task being used to copy a second set of data objects in the first storage cluster to a third storage cluster, and the second copy subtask being used to copy a portion of second data objects in the second set of data objects to the third storage cluster; a first packaging unit, configured to, when there are multiple first-class data objects with a data volume less than or equal to a first data volume threshold in the second data objects copied by the second copy subtask, package at least a portion of the multiple first-class data objects, and copy the packaged data packets to the third storage cluster; and a second splitting unit, configured to, when there are second-class data objects with a data volume greater than a second data volume threshold in the second data objects copied by the second copy subtask and the second-class data objects do not exist in the third storage cluster, split the second-class data objects into multiple sub-data objects, and copy the multiple sub-data objects respectively to the third storage cluster.

[0143] It should be noted that the above modules can be implemented through software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0144] According to another aspect of the embodiments of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the steps of any of the above-described method embodiments through the computer program. In an exemplary embodiment, the electronic device may further comprise a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0145] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail here.

[0146] According to another aspect of the embodiments of the present application, a computer program product is provided. The computer program product includes a computer program / instruction. The computer program / instruction includes a program code for executing the method shown in the flowchart.

[0147] Figure 5 The following schematically shows a block diagram of a computer system structure of an electronic device for implementing an embodiment of the present application. Figure 5 As shown, computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to programs stored in ROM 502 or programs loaded from storage unit 508 into RAM 503. Various programs and data required for system operation are also stored in random access memory 503. CPU 501, read-only memory 502, and random access memory 503 are connected to each other via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0148] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 509 including a network interface card such as a local area network card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0149] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit 501, the various functions provided by the embodiments of the present application are executed. The above-mentioned serial numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0150] It should be noted that Figure 5 The computer system 500 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented using a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented using program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be performed in an order different from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software. The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for copying objects across storage clusters, characterized in that: include: When the current node is a task control node of the first storage cluster, obtaining a first copy task to be executed, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster; Splitting the first copy task into a group of copy subtasks based on a task type of the first copy task, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy some of the first data objects in the first data object set to the second storage cluster; The group of copy subtasks is distributed to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data object to the second storage cluster according to the copy subtasks in the distributed group of copy subtasks.

2. The method according to claim 1, characterized in that Task information of the copy task issued by the service platform to the first storage cluster is stored in the first database of the first storage cluster; The step of obtaining a first copy task to be executed when the current node is a task control node of the first storage cluster includes: In a case where the current node is the task control node of the first storage cluster, searching the first database for copy tasks whose task status belongs to a specified state set, wherein the specified state set includes an unexecuted state and an incomplete split state, and the incomplete split state is a state in which the corresponding copy task has started to be split into copy subtasks but the splitting has not been completed; When a copy task whose task status belongs to the specified state set is searched, the copy task whose task status belongs to the specified state set and whose creation time is the earliest is determined as the first copy task to be executed, wherein, when the task status of the first copy task is the unexecuted state, the task status of the first copy task is modified to the split incomplete state.

3. The method according to claim 2, characterized in that The step of splitting the first copy task into a group of copy subtasks based on the task type of the first copy task includes: When the task status of the first copy task is the splitting incomplete state, determining a current splitting position based on the task tag of the first copy task, wherein the task tag of the first copy task is used to indicate an object position of a first first data object in the first data object set that has not been split into a copy subtask; Based on the task type of the first copy task, the first data object located after the current split position in the first data object set is split into copy subtasks to obtain the set of copy subtasks.

4. The method according to claim 2, characterized in that The distributing the set of copy subtasks to multiple task execution nodes of the first storage cluster includes: Writing task information of the copy subtasks in the set of copy subtasks into a second database of the first storage cluster, so that a task execution node among the plurality of task execution nodes searches for the copy subtasks from the second database for execution, thereby completing the distribution of the set of copy subtasks, wherein the first database and the second database are the same database or different databases; The copy subtasks in the group of copy subtasks are assigned corresponding task execution nodes by the current node, or are selected for execution in sequence according to the order in which they are searched by the task execution nodes in the multiple task execution nodes.

5. The method according to claim 4, characterized in that After writing the task information of the copy subtasks in the set of copy subtasks into the second database, the method further includes: When the task information of all copy subtasks in the group of copy subtasks has been written into the second database, updating the task status of the first copy task to a distribution completion status, wherein the distribution completion status is a status in which the task information of the copy subtasks into which the corresponding copy task is split has been written into the second database; Periodically retrieving the task status of the copy subtasks in the set of copy subtasks; In the case where there is a first copy subtask in the group of copy subtasks whose task status is in the executing state, determining the task execution node corresponding to the first copy subtask based on the execution node information of the first copy subtask, wherein the execution node information is used to indicate the task execution node that executes the corresponding copy subtask; in the case where the task execution node corresponding to the first copy subtask is not in the surviving state, clearing the execution node information of the first copy subtask and updating the task status of the first copy subtask to the unexecuted state; If there is no copy subtask in the group of copy subtasks whose task status is one of the unexecuted state and the executing state, the task status of the first copy task is updated to the completed state.

6. The method according to claim 1, characterized in that The step of splitting the first copy task into a group of copy subtasks based on the task type of the first copy task includes: If the task type of the first copy task is a batch copy task in which data objects to be copied are indicated by an object list, the data objects indicated by the object list corresponding to the first copy task are divided into groups of K data objects, and corresponding copy subtasks are generated for the divided data object groups to obtain the group of copy subtasks, where K is a positive integer greater than or equal to 2; When the task type of the first copy task is a whole-bucket copy task that copies data objects in a specified bucket, the data objects in the specified bucket corresponding to the first copy task are listed in batches of M data objects, and corresponding copy subtasks are generated for the data objects listed in different batches, to obtain the set of copy subtasks, where M is a positive integer greater than or equal to 2; In the case where the task type of the first copy task is a specified prefix copy task that copies data objects with a specified prefix, the data objects with the specified prefix corresponding to the first copy task are listed in batches with N data objects as a batch, and corresponding copy subtasks are generated for the data objects listed in different batches to obtain the set of copy subtasks, where N is a positive integer greater than or equal to 2.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When the current node is a task execution node of the first storage cluster, obtaining second copy subtasks distributed by the task control node of the first storage cluster and split into the second copy task, wherein the second copy task is used to copy the second data object set in the first storage cluster to a third storage cluster, and the second copy subtasks are used to copy part of the second data objects in the second data object set to the third storage cluster; If there are multiple first-category data objects whose data volume is less than or equal to the first data volume threshold among the second data objects copied by the second copy subtask, packing at least some of the multiple first-category data objects, and copying the resulting packed data packets to the third storage cluster; If there is a second type of data object with a data volume greater than a second data volume threshold in the second data object copied by the second copy subtask, and the second type of data object does not exist in the third storage cluster, the second type of data object is split into multiple sub-data objects, and the multiple sub-data objects are copied to the third storage cluster respectively.

8. A cross-object storage cluster copy device, characterized in that: include: A first acquiring unit is configured to acquire, when the current node is a task control node of the first storage cluster, a first copy task to be executed, wherein the first copy task is used to copy a first data object set of the first storage cluster to a second storage cluster; a first splitting unit, configured to split the first copy task into a group of first copy subtasks based on a task type of the first copy task, wherein the task type of the first copy task is used to locate the first data object set, and the copy subtasks in the group of copy subtasks are used to copy some of the first data objects in the first data object set to the second storage cluster; The first execution unit is used to distribute the set of copy subtasks to multiple task execution nodes of the first storage cluster, so that the task execution nodes among the multiple task execution nodes copy the corresponding first data object to the second storage cluster according to the copy subtasks in the distributed set of copy subtasks.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

Images