Method, apparatus and computer program product for data management

Abstract

Embodiments of the present disclosure provide methods, devices, and computer program products for data management. A method for data management includes: obtaining a seed file related to the download of a target file from a storage system, the target file is stored in the storage system, the seed file includes at least a plurality of data summaries of a plurality of data blocks of the target file; generating metadata for a plurality of data blocks based on the seed file, the metadata being in a format supported by a unified management system, the unified management system being configured for data access across the storage system and at least one other storage system; and storing the metadata for data block-level access to the plurality of data blocks via the unified management system. In this way, metadata in a format supported by the unified management system can be obtained to support a desired data access method.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of data storage, and more particularly, to methods, electronic devices, and computer program products for data management. Background Art

[0002] Currently, as business and application requirements change, customers often need to store data in different storage systems, especially different cloud storage platforms. Since different storage service providers usually provide different data access interfaces, customers need to interact with different interfaces. This will be troublesome and time-consuming. In this case, providing a unified management architecture for the storage environment is a trend. The unified management architecture allows customers to manage different storage systems through a single interface, realizing data storage, access and management across multiple storage systems. Summary of the invention

[0003] An embodiment of the present disclosure provides a data management solution.

[0004] In a first aspect of the present disclosure, a method for data management is provided. The method includes: obtaining a seed file related to downloading of a target file from a storage system, the target file being stored in the storage system, the seed file comprising at least a plurality of data summaries of a plurality of data blocks of the target file; generating metadata for the plurality of data blocks based on the seed file, the metadata being in a format supported by a unified management system, the unified management system being configured for data access across the storage system and at least one other storage system; and storing the metadata for data block-level access to the plurality of data blocks via the unified management system.

[0005] In a second aspect of the present disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, and when the instructions are executed by the processor, the device performs an action, the action including: obtaining a seed file related to downloading a target file from a storage system, the target file being stored in the storage system, the seed file at least including multiple data summaries of multiple data blocks of the target file; generating metadata for multiple data blocks based on the seed file, the metadata being in a format supported by a unified management system, the unified management system being configured for data access across the storage system and at least one other storage system; and storing the metadata for data block-level access to the multiple data blocks via the unified management system.

[0006] In a third aspect of the present disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes computer-executable instructions, which, when executed, cause a device to obtain a seed file related to the download of a target file from a storage system, the target file being stored in the storage system, the seed file comprising at least a plurality of data summaries of a plurality of data blocks of the target file; generate metadata for the plurality of data blocks based on the seed file, the metadata being in a format supported by a unified management system, the unified management system being configured for data access across the storage system and at least one other storage system; and store the metadata for data block-level access to the plurality of data blocks via the unified management system.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the disclosure, nor is it intended to limit the scope of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present disclosure.

[0009] Figure 1 A schematic block diagram showing an example storage environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic diagram showing an example process of implementing data management in a unified management architecture according to an embodiment of the present disclosure;

[0011] Figure 3 An example of a seed file according to some embodiments of the present disclosure is shown;

[0012] Figure 4 shows an example of metadata in a hash tree structure according to some embodiments of the present disclosure;

[0013] Figure 5 A flowchart illustrating a process for data management according to some embodiments of the present disclosure; and

[0014] Figure 6 A block diagram of an example device that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0015] The principles of the present disclosure will be described below with reference to several example embodiments shown in the accompanying drawings. Although preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way.

[0016] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] As mentioned above, providing a unified management architecture for storage environments is a trend. The unified management architecture allows customers to manage different storage systems through a single interface, and implements data storage, access, and management across multiple storage systems.

[0018] Figure 1 1 is a schematic block diagram of an example storage environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 In the environment 100, a unified management system 110 is deployed to manage multiple storage systems 120-1, 120-2, 120-3, ..., 120-N (hereinafter collectively or individually referred to as storage system 120), where N is an integer greater than or equal to 2.

[0019] Multiple storage systems 120 may be of different types and may come from different providers. Multiple storage systems 120 have certain data storage and computing capabilities. In some embodiments, multiple storage systems 120 may include one or more cloud storage services, which may be provided by one or more cloud service providers. In some embodiments, multiple storage systems 120 may also include one or more local or remote file systems (FS), databases (DB), etc.

[0020] The multiple storage systems 120 may have their own data storage and access mechanisms, and therefore need to be accessed via their own data access interfaces, such as different application programming interfaces (APIs). The deployment of the unified management system 110 enables the client 130 to achieve unified access to the multiple storage systems 120 via a unified interface with the unified management system 110. In operation, the client 130 may store the data that it wishes to store to each storage system 120 via the unified management system 110. The client 130 may also directly send a data access request (read, write, modify data, etc.) to the unified management system 110, and the unified management system 110 may initiate corresponding data access operations to the multiple storage systems 120.

[0021] The unified management system 110 and each storage system 120 may be communicatively coupled via a wired network, a wireless network, or a combination of the two. The client 130 may also be communicatively coupled with the unified management system 110 via a wired network, a wireless network, or a combination of the two.

[0022] An important aspect of enabling data access to a storage system is the maintenance of metadata associated with the stored data. In this article, metadata refers to information sufficient to provide access to the associated data. From the perspective of metadata management, the unified management system 110 can be implemented in a centralized architecture or a distributed architecture. In the implementation of the centralized architecture, the unified management system 110 stores and manages metadata stored in multiple storage systems 120 in a centralized manner. In the implementation of the distributed architecture, the unified management system 110 can be implemented in multiple computing devices (e.g., multiple edge computing nodes or servers), where the storage and management of metadata is distributed to multiple computing devices to achieve.

[0023] Regardless of the implementation method, in general, the unified management system 110 has corresponding metadata format requirements. In some implementations, the unified management system 110 can provide data access based on the data block level, allowing different data blocks divided from the same file to be stored across multiple storage systems, and allowing the client 130 to perform access operations on a data block by data block basis. This can achieve more flexible data access and more efficient storage space utilization.

[0024] It should be understood that Figure 1 The storage environment shown is only an example. In actual applications, there may be more other components in the storage environment, or these components may be arranged in other ways. For example, in some embodiments, although shown as separate components, one or more storage systems may be local storage systems of the client.

[0025] The deployment of a unified management system can bring many advantages to client data management and access. Due to the need for metadata management, if data is stored in a storage system via a unified management system, the unified management system can often effectively construct metadata for the new data. However, for data that has been previously stored in some storage systems, since the data is not stored through a unified management architecture, there is a lack of metadata for accessing the data. How to manage existing data through a unified management architecture is a problem.

[0026] A simple solution is to download the existing data from the storage system, and then upload and store the data again through the unified management system, so that the unified management system can build the required metadata accordingly. However, this process requires a large amount of data transmission (including uploading and downloading), which is not only time-consuming but also prone to errors. In addition, for systems such as cloud storage services, data downloading and uploading may also incur additional costs.

[0027] Therefore, it is expected to provide a more efficient solution to support the integration of stored data into a unified management architecture for management, while achieving data block-level access.

[0028] According to an embodiment of the present disclosure, a solution for implementing data in a unified management architecture across multiple storage systems is proposed. According to the solution, a seed file related to the download of a target file, such as metadata in a bit torrent (BT) format, is obtained from a storage system. The seed file includes at least a plurality of data summaries of a plurality of data blocks of the target file, so that it can be used to generate metadata in a format supported by a unified management system. The generated metadata is stored for data block-level access to the plurality of data blocks via the unified management system.

[0029] In this way, based on the metadata about the stored data that the storage system can provide, metadata in a format supported by the unified management system can be obtained through metadata conversion to support the desired data access method. Such a solution does not require the actual download and upload of the target file itself, saving time and economic costs.

[0030] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0031] Figure 2 FIG. 1 is a schematic diagram showing an example process of implementing data management in a unified management architecture according to an embodiment of the present disclosure. Figure 1 This process is illustrated based on an example unified management architecture.

[0032] In an embodiment of the present disclosure, it is assumed that the storage system 120-1 stores data at a file granularity, and the data stored therein is not stored via the unified management system 110. For example, before the unified management system 110 is deployed to implement management across multiple storage systems 120, the storage system 120-1 may have stored certain data, which is stored via an access interface specific to the storage system 120-1. For example, it is assumed that the storage system 120-1 stores a target file 202. According to an embodiment of the present disclosure, the unified management system 110 is able to create metadata for the target file 202 that has been stored in the storage system 120-1, without the need to download the target file 202 from the storage system 120-1 to the local for processing.

[0033] Specifically, the unified management system 110 requests 210 the storage system 120-1 for a seed file 204 related to the download of the target file 202, and obtains 220 the seed file 204 from the storage system 120-1. The unified management system 110 may send the request and obtain the seed file 204 via an access interface that communicates with the storage system 120-1 (e.g., by calling a corresponding API).

[0034] In order to provide a seed file for downloading a specific file, the storage system 120-1 generally supports file transfer using a peer-to-peer protocol. An example of an end-to-end protocol is the BitTorrent (BT) protocol. Many storage systems can support such a protocol, such as a storage system based on S3 (Simple Storage Service). A more specific example of a storage system based on S3 is the Amazon S3 cloud storage platform. However, it should be understood that the examples mentioned here are not restrictive. Embodiments of the present disclosure may be applied to any storage system that supports an end-to-end protocol, such as the BT protocol.

[0035] The main principle of the end-to-end protocol is to logically divide the file to be downloaded into multiple data blocks, and write the data summary of each data block into the seed file to support the multi-source file transmission mechanism. Therefore, the seed file can be used as an "index" of the file to be downloaded. In the embodiment of the present disclosure, the useful information contained in the seed file 204 can be used to effectively construct the metadata required by the unified management system 110.

[0036] Typically, a seed file includes two parts: tracker server information and file information. Tracking information mainly includes the address of the tracking server needed for downloading and the settings for the tracking server, and file information includes information related to the target file. File information may include, for example, file name, file size (e.g., in bytes), data block size (e.g., in bytes), data summary of each data block, and the like. The data summary may include a hash value of the data block. The algorithm used to calculate the hash value may be various hash algorithms, such as the SHA-1 hash function, and the like.

[0037] Figure 3 An example of a seed file 204 associated with the download of a target file 202 is shown. Assume that the target file 202 is named "debian-503-amd64-CD-1.iso". The seed file 204 includes tracker server information 305, and also includes file information, which includes a file name portion 310 (indicating the file name debian-503-amd64-CD-1.iso), a size-related portion 320 (indicating that the data block size is 262144 bytes, and the file size is 678301696 bytes), and a data summary portion 330 (indicating a hash value due to the data block). Figure 3 In the example of , hash values ​​of multiple data blocks (five in this example) of the target file 202 are indicated by ab, cd, ef, gh, and ij, respectively, but it should be understood that the actual hash value can be represented as a binary string.

[0038] Note that, according to the seed file 204, it can be seen that the target file 202 is logically divided into multiple data blocks, but in fact, the data in the target file 202 may not be stored in data blocks in the storage system 120-1, and data block-level access is not supported.

[0039] It should be understood that Figure 3 The seed file 204 shown is for illustrative purposes only and does not have any limitation on the embodiments of the present disclosure. The target file 202 may be logically divided into more or fewer data blocks according to other sizes, and the seed file 204 may also contain other information.

[0040] After obtaining the seed file 204 from the storage system 120-1, the unified management system 110 generates metadata 206 for multiple data blocks of the target file 202 based on the seed file 204. The unified management system 110 may have a metadata format supported by itself, so useful information can be extracted from the seed file 204 to construct metadata 206 in a corresponding format. Such metadata 206 is stored for subsequent data block-level access to the target file 202. As mentioned above, the seed file 204 includes at least the corresponding data summaries of the multiple data blocks. The unified management system 110 can use this information to generate the required metadata.

[0041] The size of the seed file 204 is usually very small, especially compared to the size of the target file 202. Therefore, according to the embodiments of the present disclosure, the existing data in the storage system can be integrated into the unified management system at a lower cost (for example, only a small amount of data needs to be downloaded from the storage system 120-1) and in a more efficient and faster manner.

[0042] When generating metadata 206, unified management system 110 extracts corresponding data digests of multiple data blocks of target file 202 from seed file 202, and determines corresponding storage locations of multiple data blocks in storage system 120-1 based at least on these data digests (e.g., hash values ​​of data blocks). Unified management system 110 determines metadata 206 as being able to indicate a mapping of multiple data digests to multiple storage locations. It should be understood that, based on seed file 202, and in particular on multiple data digests, the storage location of each data block can be determined by following the provisions of BT protocol. This is well known to those skilled in the art, especially those skilled in the art of end-to-end data transmission.

[0043] In some embodiments, if the unified management system 110 supports organizing metadata 206 of multiple data blocks in a hierarchical format, then for the target file 202, the unified management system 110 can create a hierarchical structure based on multiple data summaries when generating metadata 206. In the implementation where multiple data summaries include hash values ​​of multiple data blocks, such a hierarchical structure includes a hash tree, such as a Merkle tree, or also referred to as a Merkle directed acyclic graph (DAG). In such a hierarchical structure, multiple leaf nodes are used to indicate multiple data summaries, and one or more levels can continue to be constructed on multiple leaf nodes, with nodes of each level pointing to multiple nodes of the next level and generating additional data summaries based on the data summaries indicated by the pointed nodes.

[0044] Figure 4The metadata 206 in a hash tree structure is shown. In this example, the metadata 206 includes a hash tree 400. The leaf nodes 421 to 425 of the hash tree 400 respectively indicate multiple hash values ​​of multiple data blocks of the target file 202, which are from the data summary part 330 of the seed file 204. Nodes 411 and 412 are also included above these leaf nodes 421 to 425. Node 411 is constructed as a parent node of leaf nodes 421, 422, and 423, indicating a hash value "kl", which is generated based on the hash values ​​"ab", "cd", and "ef" of leaf nodes 421, 422, and 423. Leaf node 412 is constructed as a parent node of leaf nodes 424 and 425, indicating a hash value "mn", which is generated based on the hash values ​​"gh" and "ij" of leaf nodes 424 and 425. The hash tree 400 further includes a root node 401 , which is constructed as a parent node of the nodes 411 and 412 , indicating a hash value “op” generated based on the hash values ​​“kl” and “mn” of the nodes 411 and 412 .

[0045] The hash values ​​of the leaf nodes 421 to 425 of the hash tree 400 are mapped to the storage locations of the data blocks of the target file 202. In such a mapping, the hash value and the storage location can be considered as a <key, value> pair, and each hash value can be indexed to the corresponding storage location, thereby realizing data access.

[0046] In addition to the hash value, other nodes of the hash tree 400 may also indicate other metadata. For example, Figure 4 The metadata part 430 of the node 412 is shown, which includes: a type field 432 indicating that the type of the node 412 is a linker; and a payload field 434, which includes the identifier (id) of the node and the hash values ​​of the nodes (links) to which it is linked, namely gh and ij. Although not shown, the node 411 also indicates a similar metadata part.

[0047] The metadata portion 440 of the root node of the hash tree 400 also similarly includes a type field 442 indicating that the type of the node 401 is a file node; and a payload field 444, which includes the file identification information 445 of the target file 202 (e.g., the file name debian-503-amd64-CD-1.iso) and the hash values ​​of the nodes (links) linked to, i.e., kl and mn. The file identification information 445 is extracted from the file name portion 310 of the seed file 204. Although not shown, the payload portion 444 may also include other information that can be extracted from the seed file 204, such as the file size, the data block size, and other information contained in the seed file 204. This information can be used for other purposes in the subsequent data access process for the target file 202. For example, information such as the file size, the data block size, etc. can be used to verify the read data.

[0048] By constructing Figure 4 Such a hierarchical structure can start from the root node and access one or more data blocks of the target file 202 as needed to implement data access at the data block level.

[0049] In some embodiments, in addition to the hierarchical structure (e.g. Figure 4 The information indicated by the other nodes can be stored as data blocks in the storage system 120 managed by the unified management system 110. For example, Figure 2 The unified management system 110 is shown storing 230 some metadata parts 206-1, 206-2, and 206-3 of the metadata 206 in the storage systems 120-2, 120-3, and 120-N. The metadata parts 206-1, 206-2, and 206-3 may correspond to the metadata parts indicated by the nodes 401, 411, and 412 of the hash tree 400, including metadata parts 440 and 430, respectively.

[0050] In an embodiment where the metadata portions indicated by each node of the hierarchical node are stored in a storage system, the data summaries indicated by each node can be mapped to the storage locations of these metadata portions. As mentioned above, the leaf nodes in the hierarchical structure are mapped to the storage locations of each data block of the target file 202 in the storage system 120-1. In this way, the hash values ​​indicated by each node of the hierarchical structure (e.g., hash tree 400) are mapped to corresponding storage locations, forming multiple <key, value> pairs. Table 1 below lists the <key, value> pairs corresponding to each node of the hash tree 400.

[0051] Table 1 <key, value> pairs

[0052] key value ab https: / / my-precious-bucket.s3.amazonaws.com / ... / data:bt-part-1 cd https: / / my-precious-bucket.s3.amazonaws.com / ... / data:bt-part-2 ef https: / / my-precious-bucket.s3.amazonaws.com / ... / data:bt-part-3 gh https: / / my-precious-bucket.s3.amazonaws.com / ... / data:bt-part-4 lj https: / / my-precious-bucket.s3.amazonaws.com / ... / data:bt-part-5 kl http: / / mystorageaccount.blob.core.windows.net / mycontainer / kl mn https: / / my-precious-bucket.s3.amazonaws.com / ... / mn op IPFS-NODE-ID

[0053] It should be understood that Figure 4 The storage locations of the data blocks listed in the hash tree 400 and Table 1 and the hash values ​​of the parent nodes of the hash tree (e.g., "kl", "mn", "op") are merely examples and are not limitations of the embodiments of the present disclosure. The hash tree 400 may have more hierarchical structures, and each parent node may point to more child nodes. The storage locations indicated by the hash values ​​"kl", "mn", "op" in Table 1 may vary depending on the specific storage requirements for the metadata portion. For example, in some examples, Figure 4 The metadata parts 430, 440, etc. shown can be stored centrally rather than distributed across multiple storage systems. In some embodiments, depending on the specific storage locations of these data blocks or metadata parts, the unified management system 110 may need to request access to the data blocks or metadata parts through an access interface with the corresponding storage system 120.

[0054] After the metadata 206 is constructed and stored, the unified management system 110 can access one or more data blocks of the target file 202 based on the metadata 206. By constructing the metadata, the unified management system 110 can perform data block-level access to the target file 202 stored in the storage system 120-1. Thus, the client 130 can request to download one or more data blocks without downloading the entire file. In addition, data block-level access can further improve the security of data access, support end-to-end data transmission and multi-threaded downloading, and improve data access efficiency.

[0055] Specifically, during the data access process, if the client 130 desires to access a data block in the target file 202, the unified management system 110 can extract a data summary (e.g., a hash value) of the data block from the request. Then, by comparing the obtained hash value with the hash tree (e.g., Figure 4 The matched hash value is determined by comparing the hash values ​​of each node in the hash tree 400). The matching of hash values ​​can start from the root node and go down layer by layer. When comparing with each node, the unified management system 110 can determine the metadata portion indicated by each node through the mapping of Table 1. If the hash value in the request matches the hash value of a leaf node of the hash tree, the unified management system 110 can determine the storage location of the data block associated with the leaf node from Table 1, thereby performing access to the data block (e.g., reading, updating, moving, deleting, etc.) based on the storage location.

[0056] Figure 5 FIG. 5 is a flowchart of a process 500 for data management according to some embodiments of the present disclosure. The process 500 may be implemented in Figure 2For ease of explanation, refer to Figure 2 The process 500 is described below.

[0057] In box 510, the unified management system 110 obtains a seed file related to the download of the target file from the storage system. The target file is stored in the storage system, and the seed file includes at least a plurality of data summaries of a plurality of data blocks of the target file. In box 520, the unified management system 110 generates metadata for the plurality of data blocks based on the seed file. The metadata is in a format supported by the unified management system, and the unified management system is configured for data access across the storage system and at least one other storage system. In box 530, the unified management system 110 stores the metadata for data block-level access to the plurality of data blocks via the unified management system.

[0058] In some embodiments, generating metadata includes: extracting multiple data digests from a seed file; determining multiple storage locations of multiple data blocks in a storage system based at least on the multiple data digests; and generating second metadata to indicate a mapping of the multiple data digests to the multiple storage locations.

[0059] In some embodiments, generating metadata also includes: creating a hierarchical structure based on multiple data summaries, multiple leaf nodes of the hierarchical structure respectively indicating multiple data summaries, and parent nodes of the multiple leaf nodes indicating additional data summaries generated based on at least two of the multiple data summaries.

[0060] In some embodiments, the plurality of data digests comprises hash values ​​of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.

[0061] In some embodiments, generating metadata further includes: extracting file identification information of the target file from the seed file; and determining a root node of the hierarchical structure based on the file identification information to map the data summary indicated by the root node to a storage location of the file identification information.

[0062] In some embodiments, the first format includes a Bit Torrent (BT) format, and wherein obtaining the seed file includes: initiating a request for the seed file of the target file to the storage system via the unified management system.

[0063] In some embodiments, process 500 further includes: in response to an access request for a target data block among the plurality of data blocks, accessing the target data block from the storage system based on the metadata.

[0064] Figure 6 1 shows a schematic block diagram of an example device 600 that can be used to implement an embodiment of the present disclosure. The device 600 can be a general computing device, an edge computing node, a server, a mainframe, etc. The device 600 can be implemented as or included in Figure 2 For example, the unified management system 110 may include one or more devices 600 .

[0065] As shown, the device 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0066] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0067] Processing unit 601 performs the various methods and processes described above, such as process 500. For example, in some embodiments, process 500 may be implemented as a computer software program or computer program product, which is tangibly contained in a machine-readable medium, such as a non-transitory computer-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by CPU 601, one or more steps of process 500 described above may be performed. Alternatively, in other embodiments, CPU 601 may be configured to perform process 500 in any other suitable manner (e.g., by means of firmware).

[0068] Those skilled in the art should understand that the various steps of the method disclosed above can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present disclosure is not limited to any specific combination of hardware and software.

[0069] It should be understood that although several devices or sub-devices of the device are mentioned in the detailed description above, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided into multiple devices to be embodied.

[0070] The above description is only an optional embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for data management, comprising: obtaining, from an object storage system, a first set of metadata associated with downloading of a target file via a peer-to-peer protocol, the target file being stored on the object storage system in a manner accessible at a file-level granularity, the first set of metadata comprising a plurality of data digests corresponding to a plurality of data blocks, the target file being logically partitioned into the plurality of data digests for multi-source delivery to a client device according to the peer-to-peer protocol; generating a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system, the unified management system being configured to provide a unified interface for access by one or more client devices to a plurality of different storage systems including the object storage system; storing the second set of metadata in the unified management system to support access to portions of the target file at a block-level granularity by the one or more client devices; as well as A request is received from a client device among the one or more client devices to select, via the unified management system, an individual data block among the plurality of data blocks.

2. The method of claim 1 , wherein generating the second set of metadata comprises: extracting the plurality of data summaries from the first set of metadata; determining a plurality of storage locations of the plurality of data blocks in the object storage system based at least on the plurality of data digests; as well as Additional metadata is generated to indicate a mapping of the plurality of data digests to the plurality of storage locations.

3. The method of claim 2, wherein generating the second set of metadata further comprises: A hierarchical structure is created based on the multiple data digests, multiple leaf nodes of the hierarchical structure respectively indicate the multiple data digests, and parent nodes of the multiple leaf nodes indicate additional data digests generated based on at least two of the multiple data digests.

4. The method of claim 3, wherein the plurality of data digests comprise hash values ​​of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.

5. The method of claim 3, wherein generating the second set of metadata further comprises: extracting file identification information of the target file from the first set of metadata; as well as A root node of the hierarchical structure is determined based on the file identification information to map the data digest indicated by the root node to a storage location of the file identification information.

6. The method of claim 1, wherein obtaining the first set of metadata comprises: Initiating a request for the first set of metadata for the target file to the object storage system via the unified management system; The first set of metadata includes a seed file.

7. The method of claim 1 , wherein a first group of one or more data blocks of the target file are stored in the object storage system, and a second group of one or more data blocks are stored in at least one other storage system among the multiple different storage systems, the second group of one or more data blocks being different from the first group of one or more data blocks.

8. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, wherein when the instructions are executed by the processor, the electronic device performs actions, the actions comprising: obtaining, from an object storage system, a first set of metadata associated with downloading of a target file via a peer-to-peer protocol, the target file being stored on the object storage system in a manner accessible at a file-level granularity, the first set of metadata comprising a plurality of data digests corresponding to a plurality of data blocks, the target file being logically partitioned into the plurality of data digests for multi-source delivery to a client device according to the peer-to-peer protocol; generating a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system, the unified management system being configured to provide a unified interface for access by one or more client devices to a plurality of different storage systems including the object storage system; storing the second set of metadata in the unified management system to support access to portions of the target file at block-level granularity by the one or more client devices; and A request is received from a client device among the one or more client devices to select, via the unified management system, an individual data block among the plurality of data blocks.

9. The electronic device of claim 8, wherein generating the second set of metadata comprises: extracting the plurality of data summaries from the first set of metadata; determining a plurality of storage locations of the plurality of data blocks in the object storage system based at least on the plurality of data digests; as well as Additional metadata is generated to indicate a mapping of the plurality of data digests to the plurality of storage locations.

10. The electronic device of claim 9, wherein generating the second set of metadata further comprises: A hierarchical structure is created based on the multiple data digests, multiple leaf nodes of the hierarchical structure respectively indicate the multiple data digests, and parent nodes of the multiple leaf nodes indicate additional data digests generated based on at least two of the multiple data digests. 11 . The electronic device of claim 10 , wherein the plurality of data digests comprise hash values ​​of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.

12. The electronic device of claim 10, wherein generating the second set of metadata further comprises: extracting file identification information of the target file from the first set of metadata; as well as A root node of the hierarchical structure is determined based on the file identification information to map the data digest indicated by the root node to a storage location of the file identification information.

13. The electronic device according to claim 8, wherein acquiring the first set of metadata comprises: Initiating a request for the first set of metadata for the target file to the object storage system via the unified management system; The first set of metadata includes a seed file.

14. An electronic device according to claim 8, wherein a first group of one or more data blocks of the target file are stored in the object storage system, and a second group of one or more data blocks are stored in at least one other storage system among the multiple different storage systems, and the second group of one or more data blocks are different from the first group of one or more data blocks.

15. A computer program product tangibly stored on a non-transitory computer readable medium and comprising computer executable instructions that when executed cause a device to: obtaining, from an object storage system, a first set of metadata associated with downloading of a target file via a peer-to-peer protocol, the target file being stored on the object storage system in a manner accessible at a file-level granularity, the first set of metadata comprising a plurality of data digests corresponding to a plurality of data blocks, the target file being logically partitioned into the plurality of data digests for multi-source delivery to a client device according to the peer-to-peer protocol; generating a second set of metadata based on the first set of metadata, the second set of metadata being in a format supported by a unified management system, the unified management system being configured to provide a unified interface for access by one or more client devices to a plurality of different storage systems including the object storage system; storing the second set of metadata in the unified management system to support access to portions of the target file at a block-level granularity by the one or more client devices; as well as A request is received from a client device among the one or more client devices to select, via the unified management system, an individual data block among the plurality of data blocks.

16. The computer program product of claim 15, wherein generating the second set of metadata comprises: extracting the plurality of data summaries from the first set of metadata; determining a plurality of storage locations of the plurality of data blocks in the object storage system based at least on the plurality of data digests; as well as Additional metadata is generated to indicate a mapping of the plurality of data digests to the plurality of storage locations.

17. The computer program product of claim 16, wherein generating the second set of metadata further comprises: A hierarchical structure is created based on the multiple data digests, multiple leaf nodes of the hierarchical structure respectively indicate the multiple data digests, and parent nodes of the multiple leaf nodes indicate additional data digests generated based on at least two of the multiple data digests.

18. The computer program product of claim 17, wherein the plurality of data digests comprise hash values ​​of the plurality of data blocks, and wherein the hierarchical structure comprises a hash tree.

19. The computer program product of claim 17, wherein generating the second set of metadata further comprises: extracting file identification information of the target file from the first set of metadata; as well as A root node of the hierarchical structure is determined based on the file identification information to map the data digest indicated by the root node to a storage location of the file identification information.

20. The computer program product of claim 15, wherein obtaining the first set of metadata comprises: Initiating a request for the first set of metadata for the target file to the object storage system via the unified management system; The first set of metadata includes a seed file.

21. The computer program product of claim 15, wherein a first group of one or more data blocks of the target file are stored in the object storage system, and a second group of one or more data blocks are stored in at least one other storage system of the plurality of different storage systems, the second group of one or more data blocks being different from the first group of one or more data blocks.

Images