Object downloading method, device and system, cluster and medium

Abstract

The present application provides an object downloading method, device and system, a cluster and a medium. The method comprises: a first client sends to an object storage service on a first server a first seed acquisition request, the first seed acquisition request being used for requesting a seed file of a target object; on the basis of the first seed acquisition request, the first server sends to a first client the seed file; on the basis of the seed file, the first client sends to a second server a first multi-source downloading provider acquisition request, the first multi-source downloading provider acquisition request being used for requesting a list of at least one client providing the downloading of the target object; on the basis of the first multi-source downloading provider acquisition request, the second server sends to the first client a first multi-source downloading provider list; and the first client receives a first part of the target object sent by the first server and receives a second part of the target object sent by the second client, so as to increase the downloading speed.

Description

Object downloading method, device, system, cluster and medium

[0001] This application claims priority to the Chinese patent application with application number 202310876409.9 filed with the State Intellectual Property Office of China on July 17, 2023, and priority to the Chinese patent application with the invention name “A method, device and other equipment for object downloading”. This application claims priority to the Chinese patent application with application number 202311470739.4 filed with the State Intellectual Property Office of China on November 6, 2023, and priority to the Chinese patent application with the invention name “Object downloading method, device, system, cluster and medium”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of storage, and in particular to an object downloading method, device, system, cluster and medium. Background Art

[0003] With the rapid development of the internet, object storage has been widely adopted in various internet scenarios, providing massive, secure, low-cost, and highly reliable unstructured storage services. However, current object storage systems can only access the same object by initiating multiple requests for the same object. This can place significant pressure on object storage servers. Failure of the object storage server can result in an inability to download objects, or even object loss. Furthermore, to ensure a high user experience, service providers must incur significant costs. Consequently, they are unable to provide high-performance, low-cost support for scenarios such as live streaming on demand, concurrent image building for cloud services, and AI training.

[0004] Summary of the Invention

[0005] The present application provides an object downloading method, device, system, cluster and medium, which can download target objects from multiple download sources, thereby improving the download speed.

[0006] In a first aspect, a method for downloading an object is provided. The method is applied to an object storage system. The object storage system includes a first client, a second client, a first server, and a second server. The first server deploys an object storage service, and the first server and the second server are the same server or different servers.

[0007] The first client sends a first seed retrieval request to the object storage service on the first server. In response, the first server receives the first seed retrieval request sent by the first client. The first seed retrieval request is used to request a seed file for a target object. The target object is an object stored on the first server. Based on the first seed retrieval request, the first server sends the seed file to the first client. In response, the first client receives the seed file sent by the first server based on the first seed retrieval request.

[0008] The second client sends a second seed retrieval request to the object storage service on the first server. In response, the first server receives the second seed retrieval request sent by the second client. The second seed retrieval request is used to request a seed file for a target object. Based on the second seed retrieval request, the first server sends the seed file to the second client. In response, the second client receives the seed file sent by the first server based on the second seed retrieval request.

[0009] The first client sends a first multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server receives the first multi-source download provider acquisition request sent by the first client based on the seed file. Based on the first multi-source download provider acquisition request, the second server sends a first multi-source download provider list to the first client. Accordingly, the first client receives the first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request. The first multi-source download provider acquisition request is used to request a list of at least one client that is authorized to download the target object. For example, this may include the second client.

[0010] The second client sends a second multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server receives the second multi-source download provider acquisition request sent by the second client based on the seed file. Based on the second multi-source download provider acquisition request, the second server sends a second multi-source download provider list to the second client. Accordingly, the second client receives the second multi-source download provider list sent by the second server based on the second multi-source download provider acquisition request. The second multi-source download provider acquisition request is used to request a list of at least one client that is authorized to download the target object. For example, this may include the first client.

[0011] The first client receives a first part of the target object sent by the first server, and the second client receives a second part of the target object sent by the first server.

[0012] The first client sends a first download request to the second client based on the first multi-source download provider list. Correspondingly, the second client receives the first download request sent by the first client based on the first multi-source download provider list. The second client sends a second portion of the target object to the first client. Correspondingly, the first client receives the second portion of the target object sent by the second client.

[0013] In the above scheme, the object is divided into multiple parts. After the second client downloads the second part of the object, the first client can download the first part of the object from the first server and download the second part of the object from the second client. Therefore, downloading objects from multiple channels is achieved, which improves the speed of object downloading. Moreover, even if the first server fails, the object can still be downloaded from other clients, reducing the risk of being unable to download the object due to a failure of the first server. At this time, there is no need to perform active-active backup of the first server, which reduces costs.

[0014] In some possible designs, to ensure data security, the first client needs to authenticate the second client before sending the first part of the target object to the second client. Similarly, the second client needs to authenticate the first client before sending the second part of the target object to the first client. If the first client is successfully authenticated, the second server sends the first token to the first client. If the second client is successfully authenticated, the second server sends the second token to the second client.

[0015] The first client sends the first token to the second client. In response, the second client receives the first token sent by the first client. The second client sends the first token to a second server. In response, the second server receives the first token sent by the second client. The second server verifies the first token and, if the verification is successful, sends a first verification success message to the second client. In response, the second client receives the first verification success message sent by the second server. Only after receiving the first verification success message will the second client send the second portion of the target object to the first client.

[0016] The second client sends the second token to the first client. In response, the first client receives the second token sent by the second client. The first client sends the second token to the second server. In response, the second server receives the second token sent by the first client. The second server verifies the second token and, if the verification is successful, sends a second verification success message to the second client. In response, the second client receives the second verification success message sent by the second server. Only after receiving the second verification success message will the second client send the first portion of the target object to the first client.

[0017] In the above solution, because the relationship between the client and the first server in previous object storage systems is one-to-one, the relationship is relatively simple and does not require complex verification. However, in this application, the relationship between the internal members of the system is very complex. In addition to downloading data from the first server, the client can also download data from other clients. Therefore, it is necessary to verify the identity of the other party before sending data.

[0018] In some possible designs, the first server is configured to grant the first client a first permission to obtain the seed file and a second permission to obtain the target object. Similarly, the first server is configured to grant the second client a first permission to obtain the seed file and a second permission to obtain the target object, wherein the first permission is lower than the second permission.

[0019] In the above solution, because the confidentiality level of the data contained in the seed file is lower than that of the target object, the first permission for obtaining the seed may be lower than the second permission for obtaining the target object.

[0020] In some possible designs, the first server includes a metadata server and a user data server. The target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server. At this time, after the first server receives the first seed acquisition request, it obtains the metadata of the target object from the metadata server. Then, the first server generates a seed file for the target object based on the metadata of the target object. Therefore, the generated seed file includes the metadata of the target object. Therefore, when the second client sends a second seed acquisition request to the first server, the seed file sent by the first server to the second client includes the metadata of the target object. The second client only needs to download the user data of the target object, thereby effectively reducing the download delay. Here, the first server and the second server can be different servers in the same area, or different servers in different areas.

[0021] In some possible designs, the first server creates the seed file for the target object based on the seed acquisition request and sends the seed file to the first client.

[0022] In some possible designs, the second server authenticates the first client based on the first multi-source download provider acquisition request, and if the first client passes the authentication, sends the first multi-source download provider list to the first client. The second server authenticates the second client based on the second multi-source download provider acquisition request, and if the second client passes the authentication, sends the second multi-source download provider list to the second client. Here, the first multi-source download provider list and the second multi-source download provider list can be the same or different.

[0023] In a second aspect, an object storage system is provided, comprising a first client, a second client, a first server, and a second server, wherein the first server deploys an object storage service, and the first server and the second server are the same server or different servers.

[0024] The first client is configured to send a first seed retrieval request to the object storage service on the first server. Accordingly, the first server is configured to receive the first seed retrieval request sent by the first client. The first seed retrieval request is configured to request a seed file for a target object. The target object is an object stored on the first server. The first server is configured to send the seed file to the first client based on the first seed retrieval request. Accordingly, the first client is configured to receive the seed file sent by the first server based on the first seed retrieval request.

[0025] The second client is configured to send a second seed retrieval request to the object storage service on the first server. Accordingly, the first server is configured to receive the second seed retrieval request sent by the second client. The second seed retrieval request is configured to request a seed file for a target object. Based on the second seed retrieval request, the first server is configured to send the seed file to the second client. Accordingly, the second client is configured to receive the seed file sent by the first server based on the second seed retrieval request.

[0026] The first client is used to send a first multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server is used to receive the first multi-source download provider acquisition request sent by the first client based on the seed file. The second server is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request. Correspondingly, the first client is used to receive the first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request. The first multi-source download provider acquisition request is used to request a list of at least one client that downloads the target object. For example, it may include a second client.

[0027] The second client is configured to send a second multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server is configured to receive the second multi-source download provider acquisition request sent by the second client based on the seed file. The second server is configured to send a second multi-source download provider list to the second client based on the second multi-source download provider acquisition request. Accordingly, the second client is configured to receive the second multi-source download provider list sent by the second server based on the second multi-source download provider acquisition request. The second multi-source download provider acquisition request is configured to request a list of at least one client that is authorized to download the target object. For example, the list may include the first client.

[0028] The first client is used to receive a first part of the target object sent by the first server, and the second client is used to receive a second part of the target object sent by the first server.

[0029] The first client is configured to send a first download request to a second client based on the first multi-source download provider list. Accordingly, the second client is configured to receive the first download request sent by the first client based on the first multi-source download provider list. The second client is configured to send a second portion of the target object to the first client. Accordingly, the first client is configured to receive the second portion of the target object sent by the second client.

[0030] In some possible designs, the first client is configured to authenticate the second client before sending the first portion of the target object to the second client. Similarly, the second client is configured to authenticate the first client before sending the second portion of the target object to the first client. The second server is configured to send a first token to the first client if the first client is successfully authenticated, and to send a second token to the second client if the second client is successfully authenticated.

[0031] The first client is configured to send the first token to the second client. Accordingly, the second client is configured to receive the first token sent by the first client. The second client is configured to send the first token to a second server. Accordingly, the second server is configured to receive the first token sent by the second client. The second server is configured to verify the first token and, if the verification is successful, send a first verification success message to the second client. Accordingly, the second client is configured to receive the first verification success message sent by the second server. After receiving the first verification success message, the second client is configured to send the second portion of the target object to the first client.

[0032] The second client is configured to send the second token to the first client. Accordingly, the first client is configured to receive the second token sent by the second client. The first client is configured to send the second token to a second server. Accordingly, the second server is configured to receive the second token sent by the first client. The second server is configured to verify the second token and, if verification is successful, send a second verification success message to the second client. Accordingly, the second client is configured to receive the second verification success message sent by the second server. After receiving the second verification success message, the second client is configured to send the first portion of the target object to the first client.

[0033] In some possible designs, the first server is configured to grant the first client a first permission to obtain the seed file and a second permission to obtain the target object. Similarly, the first server is configured to grant the second client a first permission to obtain the seed file and a second permission to obtain the target object, wherein the first permission is lower than the second permission.

[0034] In some possible designs, the first server includes a metadata server and a user data server. The target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server. At this time, after the first server receives the first seed acquisition request, it obtains the metadata of the target object from the metadata server, and then generates a seed file for the target object based on the metadata of the target object. Therefore, the generated seed file includes the metadata of the target object. Therefore, when the second client sends a second seed acquisition request to the first server, the seed file sent by the first server to the second client includes the metadata of the target object. The second client only needs to download the user data of the target object, so the download delay can be effectively reduced. Here, the first server and the second server can be different servers in the same area, or different servers in different areas.

[0035] In some possible designs, the first server is used to create the seed file for the target object based on the seed acquisition request and send the seed file to the first client.

[0036] In some possible designs, the second server is configured to authenticate the first client based on the first multi-source download provider acquisition request, and if the first client is successfully authenticated, send the first multi-source download provider list to the first client. The second server is configured to authenticate the second client based on the second multi-source download provider acquisition request, and if the second client is successfully authenticated, send the second multi-source download provider list to the second client. Here, the first multi-source download provider list and the second multi-source download provider list may be the same or different.

[0037] According to a third aspect, a first client is provided, wherein the first client belongs to an object storage system, the object storage system further includes a second client, a first server, and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first client includes:

[0038] A sending unit, configured to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object;

[0039] a receiving unit, configured to receive the seed file sent by the first server based on the first seed acquisition request;

[0040] The sending unit is configured to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with a multi-source download of the target object;

[0041] The receiving unit is configured to receive a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client;

[0042] The receiving unit is configured to receive a first portion of the target object sent by the first server, and receive a second portion of the target object sent by the second client.

[0043] In a fourth aspect, a first server is provided. The first server belongs to an object storage system. The object storage system further includes a first client, a second client, and a second server. The first server deploys an object storage service. The first server and the second server are the same server or different servers. The first server includes:

[0044] A receiving unit, configured to receive a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object;

[0045] The receiving unit is further configured to receive a second seed acquisition request sent by a second client, wherein the second seed acquisition request is used to request a seed file of the target object;

[0046] a sending unit, configured to send the seed file to the first client and the second client respectively;

[0047] The sending unit is further configured to send the first part of the target object to the first client;

[0048] The sending unit is further configured to send the second part of the target object to the second client.

[0049] In a fifth aspect, a second server is provided. The second server belongs to an object storage system. The object storage system further includes a first client, a second client, and a first server. The first server deploys an object storage service. The first server and the second server are the same server or different servers. The second server includes:

[0050] a receiving unit, configured to receive a seed file of a target object sent by the first server, and to establish a multi-source download provider list based on the seed file;

[0051] The receiving unit is further configured to receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with download services for the target object;

[0052] The sending unit is configured to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client and the first multi-source download provider list belongs to the multi-source download provider list.

[0053] In a sixth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps performed according to any one of the first client, the second client, the first server, and the second server in the first aspect are implemented.

[0054] In the seventh aspect, a computing cluster is provided, which includes multiple computer devices, each of the multiple computer devices includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the steps performed by any angle of the first server or the second server in the first aspect.

[0055] In an eighth aspect, a computer program product comprising instructions is provided, which, when executed by a computing device cluster, causes the computing device cluster to execute the steps executed by either the first server or the second server in the first aspect.

[0056] In the ninth aspect, a computer-readable storage medium is provided, which stores computer instructions. When the computer instructions are executed on a computer device, the computer device executes the steps performed by any angle of the client, the first server or the second server in the first aspect of the claim. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0058] FIG1 is a schematic diagram of the structure of an object storage provided by this application;

[0059] FIG2 is a schematic diagram of the structure of an object storage system provided by the present application;

[0060] FIG3 is a schematic diagram of an object storage server provided by the present application;

[0061] FIG4 is a flow chart of an object downloading method provided by the present application;

[0062] FIG5 is a schematic diagram of the structure of an object storage system provided by the present application;

[0063] FIG6 is a schematic diagram of the structure of a computing device provided by the present application;

[0064] FIG7 is a schematic diagram of the structure of a computing cluster provided by the present application;

[0065] FIG8 is a schematic diagram of the structure of another computing cluster provided by this application. DETAILED DESCRIPTION

[0066] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0067] Objects mainly refer to unstructured data such as movies, office documents, and images.

[0068] Object storage service (OBS) is a way to store unstructured data such as movies, office documents, and images. Object storage provides a scalable, self-managed, protected, and shareable storage solution. Object storage uses a flat address space to store data, without the hierarchy of directories and files. Thanks to this, an object storage system can store a large number of objects. As shown in Figure 1, an object can include user data, related metadata (such as size, date, owner, etc.), and other attributes of the data (such as access characteristics, etc.). Each object is identified by a unique ID, which is called an object ID. Object IDs are generated using a dedicated algorithm (such as the hash value of the data) to ensure that each object ID is unique.

[0069] Object-based storage devices (OSDs) are devices that store objects. They typically use low-cost, high-density storage devices (such as hard disks) to store objects. When more capacity is needed, more hard disks can be added.

[0070] 2 , which is a schematic diagram of the structure of an object storage system provided by the present application. As shown in FIG2 , the object storage system of this embodiment includes: multiple clients 110 , a first server 120 , and a second server 130 .

[0071] Client 110 is a software tool or library for interacting with the server. By providing an application programming interface (API) or command line tool, it enables users to perform basic operations such as uploading, downloading, deleting, copying, and renaming, as well as manage buckets and data stored in OSDs. The first client can be an OBS client. To enable multi-source downloading of objects, new multi-source download-related functions can be added to the existing OBS client functions, such as obtaining seed files and running seed files. Multi-source downloading means that in addition to downloading objects from the server, objects can also be downloaded from other clients.

[0072] First server 120 can run the OSD operating system and provide data storage, retrieval, and management services. As shown in Figure 3, first server 120, as an object storage server, provides two primary services: metadata and storage services. Upon receiving unstructured data to be stored, the object storage server divides the data into two components: user data and metadata. It then applies a dedicated algorithm to generate an object ID, which is unique based on the content of the user data. To facilitate subsequent access, the object storage server uses the metadata service to store the metadata and object ID, and uses the storage service to store the user data (object) in the object storage device. After successful storage, the stored content can be retrieved using the object ID. Because object storage is typically a cloud service, user data and metadata are often stored separately to enable object storage to automatically adjust its resource size to meet the needs of different workloads and provide sufficient flexibility and scalability to cope with fluctuating demand and traffic peaks. The user data and metadata can even be stored in remote locations, for example, user data can be stored in Beijing and metadata in Guangzhou. In order to realize multi-source downloading of objects, the first server 120 can add new functions related to multi-source downloading based on the functions of the OBS server, such as generating a seed file of the object, publishing the seed file to the second server 130, opening the client 110's permission to download the seed file, downloading the object in a multi-source download manner (detailed introduction below), etc.

[0073] Second server 130 is typically used to manage multi-source download providers. Specifically, second server 130 is used to register information about multi-source download providers. When a client 110 joins the object storage system, it registers its information with second server 130, including the internet protocol (IP) address and port number for interconnection between networks. Second server 130 maintains a list of multi-source download providers, meaning it maintains information about all participants in the multi-source download of an object. Second server 130 provides a list of multi-source download providers, meaning when a client 110 wants to download a specific file, it can request the list from second server 130. Second server 130 returns information about other clients 110 in the object storage system that are specific to the object, helping the client 110 (i.e., the downloader) establish a connection and begin downloading. Second server 130 also tracks download progress, meaning it tracks the download progress of the downloader and notifies other clients 110 of the progress, helping them select the appropriate client 110 for downloading. Second server 130 can be a tracker server with the added functionality of processing objects.

[0074] The object storage system shown in FIG2 is illustrated by taking two clients 110, a first server 120, and a second server 130 as an example. In actual applications, there may be more than three clients 110, more than two first servers 120, and more than two second servers 130. In addition, the first server and the second server may be merged into the same server.

[0075] In addition, the client, the first server, and the second server can be located in the same cloud, the client and the first server can be located in the same cloud, and the second server can be located in another cloud, the client can be located in one cloud, and the first server and the second server can be located in another cloud, or the client, the first server, and the second server can be located in different clouds.

[0076] See Figure 4, which is a flow chart of an object downloading method provided by the present application. As shown in Figure 4, the object downloading method provided by the present application includes the following steps:

[0077] Step S101: a first client sends a first seed acquisition request to a first server. Correspondingly, the first server receives the first seed acquisition request sent by the first client.

[0078] In a specific embodiment, the first client may be the client 110 in the embodiment shown in FIG2 , and the first server may be the first server 120 in the embodiment shown in FIG2 . For details, please refer to FIG2 and the introduction of related embodiments, which will not be elaborated here.

[0079] In a specific embodiment, the first seed acquisition request is used to request a seed file of a target object, which is one or more objects in the first server.

[0080] Step S102: The first server generates a seed file for the target object based on the first seed acquisition request.

[0081] In a specific embodiment, the seed file contains metadata of one or more files such as object ID, object size, object hash value, uniform resource locator (URL) of the second server, etc. Among them, the object ID is the unique identifier of the target object that the first client is going to download. The object size is the size of the data volume of the target object, for example, the size of the object is 1G bit, etc. The object hash value is the value obtained after performing a hash operation on the target object, which is used to verify the integrity of the downloaded object and ensure that the downloaded target object is consistent with the original object. The URL of the second server is used to identify and locate the second server, and the first client can access the second server according to the URL of the second server. Optionally, the seed file also includes metadata of the object.

[0082] In a specific embodiment, in addition to generating a seed file, the first server can also divide the target object into multiple parts, each of which can be called a block. When dividing the target object, the divided blocks can be blocks of equal size or blocks of unequal sizes, which is not specifically limited here. When the target object is divided into blocks of unequal sizes, the large blocks can be sent to clients with more ample bandwidth, and the small blocks can be sent to clients with less bandwidth, thereby improving the overall download effect of the entire network. In addition, the methods of dividing different objects can also be different. For example, object A can be divided into multiple blocks in units of 256 kilobits, object B can be divided into multiple blocks in units of 128 kilobits, and so on.

[0083] Step S103: The first server sends the seed file to the second server. Correspondingly, the second server receives the seed file sent by the first server.

[0084] In a specific embodiment, after the second server receives the seed file sent by the first server, it will establish a multi-source download provider list for the object in the first server. The multi-source download provider list may include the object ID, the client's IP address, etc. In addition, the multi-source download provider list may also include one or more of the client's port number, the client's upload speed, the client's download speed, the connection status (e.g., connected, disconnected, connecting), the object size, the object hash value, etc. The multi-source download provider list can be changeable, that is, when a client leaves the object storage system, the client can be deleted from the multi-source download provider list, and when a client joins the object storage system, the client can be added to the multi-source download provider list. In addition, in order to save resources, the second server may delete the multi-source download provider list if it has not been used for a long time.

[0085] Step S104: The first server sends a seed file to the first client. Correspondingly, the first client receives the seed file sent by the first server.

[0086] In a specific embodiment, the first server sends the seed file to the first client only if the first client has passed the verification. The first server verifies the first client to ensure that only trusted clients can access, use or interact with the first server, prevent unauthorized clients from accessing the first server, and protect the security and confidentiality of the seed file and the object. The verification method may include one or more of identity verification, certificate verification, and token verification. Identity verification can be to verify the serial number, media access control address (MAC) address, international mobile equipment identity (IMEI) number, etc. of the client, thereby preventing the use of forged or invalid clients for access. Certificate verification can be to verify one or more of the client's public key, client identity information, digital signature, etc., thereby ensuring the legitimacy of the client through the validity and integrity of the certificate. Token verification is to verify one or more of a specific key, access token, or client token, etc., thereby confirming the identity and authority of the client.

[0087] In a specific embodiment, in order to ensure the security of object storage, the first server will send the seed file to the first client only when the first client has the permission to obtain the seed file. The level of permission to obtain the seed file can be any one of the following: object bucket level permission, object level permission, etc. The type of permission to obtain the seed file can be public read, private, etc. Of course, in addition to the level of permission to obtain the seed file and the type of permission mentioned above, the first server can also manage the permission to obtain the seed file in a fine-grained manner through access control policies. The access control policy is a permission definition method based on the policy language, and more complex permission rules can be defined as needed. Users can freely choose the setting method, level and type of permissions as needed.

[0088] Step S105: The second client sends a second seed acquisition request to the first server. Correspondingly, the first server receives the second seed acquisition request sent by the second client.

[0089] In a specific embodiment, the second client may be the client 110 in the embodiment shown in FIG. 2 . Please refer to FIG. 2 and the introduction of related embodiments for details, and no further details will be given here.

[0090] In a specific embodiment, the second seed acquisition request is used to request a seed file of the target object.

[0091] Step S106: The first server sends the seed file to the second client. Correspondingly, the second client receives the seed file sent by the first server.

[0092] In a specific embodiment, because the first server has already generated a seed file when the first client sends a first seed acquisition request to the first server, when the second client sends a second seed acquisition request to the first server, the first server determines that the seed file has been generated for the target object. Therefore, the first server can directly send the generated seed file to the second client without generating the seed file again.

[0093] In a specific embodiment, the first server will send the seed file to the second client only when the second client also has the permission to obtain the seed file. For the content about obtaining the seed file permission, please refer to the above and will not be elaborated here. The level of the first client's permission to obtain the seed file and the level of the second client's permission to obtain the seed file can be the same or different; the type of the first client's permission to obtain the seed file and the type of the second client's permission to obtain the seed file can be the same or different; the access control policy for the first client's permission to obtain the seed file and the access control policy for the second client's permission to obtain the seed file can be the same or different.

[0094] In a specific embodiment, the first server sends the seed file to the second client only after the second client has been authenticated. The authentication method of the first server for the second client is basically similar to the authentication method of the first server for the first client, and will not be further described here.

[0095] Step S107: The first client sends a first multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server receives the first multi-source download provider acquisition request sent by the first client based on the seed file.

[0096] In a specific embodiment, since the URL of the second server is recorded in the seed file, the first client can locate the second server according to the URL of the second server, thereby achieving communication between the first client and the second server.

[0097] In a specific embodiment, the first multi-source download provider obtains a request for obtaining a list of clients for downloading the object.

[0098] In a specific embodiment, when the second server receives the first multi-source download provider acquisition request, it adds the first client to the multi-source download provider list.

[0099] Step S108: The second client sends a second multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server receives the second multi-source download provider acquisition request sent by the second client based on the seed file.

[0100] In a specific embodiment, since the seed file records the URL of the second server, the second client can locate the second server according to the URL of the second server, thereby achieving communication between the second client and the second server.

[0101] In a specific embodiment, the second multi-source download provider acquisition request is used to obtain a list of clients for downloading the object.

[0102] In a specific embodiment, when the second server receives the second multi-source download provider acquisition request, the second client is added to the multi-source download provider list.

[0103] Step S109: The second server sends the first multi-source download provider list to the first client. Correspondingly, the first client receives the first multi-source download provider list sent by the second server.

[0104] In a specific embodiment, the second server sends the multi-source download provider list to the first client only after the first client has successfully authenticated the second client. The second server authenticates the first client to ensure that only trusted clients can access, use, or interact with the second server, preventing unauthorized clients from accessing the second server and protecting the security and confidentiality of the multi-source download provider list.

[0105] In a specific embodiment, to ensure security, the second server will only send the multi-source download provider list to the first client if the first client has permission to obtain the multi-source download provider list. The permission type for obtaining the multi-source download provider list can be public read, private, or other. Of course, in addition to the permission types mentioned above, the second server can also manage the permission to obtain the multi-source download provider list in a fine-grained manner through access control policies.

[0106] In a specific embodiment, the second server may send a multi-source download provider list including all multi-source download providers or a multi-source download provider list including some multi-source download providers as the first multi-source download provider list to the first client. For example, some multi-source download providers may include multi-source download providers with relatively high upload and / or download speeds, and some multi-source download providers may include multi-source download providers other than the first client in the multi-source download provider list. In a more specific embodiment, the multi-source download provider list includes the second client.

[0107] In a specific embodiment, where security requirements are high, the second server may send a first token to the first client in addition to the list of multi-source download providers. The first token is used to authorize the first client to download the target object from the second client. The first token may be a long-term or short-term token. When the first token expires, the first client cannot download the object from the second client.

[0108] Step S110: The second server sends a second multi-source download provider list to the second client. Correspondingly, the second client receives the second multi-source download provider list sent by the second server.

[0109] In a specific embodiment, the second server sends the multi-source download provider list to the second client only after the second client has successfully authenticated the second client. The second server authenticates the second client to ensure that only trusted clients can access, use, or interact with the second server, preventing unauthorized clients from accessing the second server and protecting the security and confidentiality of the multi-source download provider list.

[0110] In a specific embodiment, in order to ensure security, the second server will send the multi-source download provider list to the second client only when the second client has the permission to obtain the multi-source download provider list. The specific introduction to the multi-source download provider list is as mentioned above and will not be repeated here. The level of the first client's permission to obtain seed files and the level of the second client's permission to obtain seed files can be the same or different; the type of the first client's permission to obtain the multi-source download provider list and the type of the second client's permission to obtain the multi-source download provider list can be the same or different; the access control policy of the first client's permission to obtain the multi-source download provider list and the access control policy of the second client's permission to obtain the multi-source download provider list can be the same or different.

[0111] In a specific embodiment, the second server can send a multi-source download provider list including all multi-source download providers or a multi-source download provider list of some multi-source download providers to the second client. For example, some multi-source download providers are multi-source download providers with relatively high upload and / or download speeds, and some multi-source download providers are other multi-source download providers in the multi-source download provider list except the second client. Because the first client downloads the multi-source download provider list first and the second client downloads the multi-source download provider list later, other clients may join or leave the multi-source download provider list in between. Therefore, the multi-source download provider list downloaded by the first client and the multi-source download provider list downloaded by the second client may be different. In a more specific embodiment, the multi-source download provider list includes the first client.

[0112] In a specific embodiment, when security requirements are relatively high, in addition to sending the multi-source download provider list to the second client, the second server also needs to send a second token to the second client. The second token is used to authorize the second client to download the object from the first client. The second token can be a long-term token or a short-term token. When the second token expires, the second client cannot download the target object from the first client. It is understood that the first token and the second token can both be long-term tokens, or short-term tokens, or the first token can be a long-term token and the second token can be a short-term token, or the first token can be a short-term token and the second token can be a long-term token.

[0113] Step S111: The first server sends the first part of the object to the first client. Correspondingly, the first client receives the first part of the object sent by the first server.

[0114] In a specific embodiment, to ensure security, the first server will only send the first part of the object to the first client if the first client has permission to obtain the object. The level of permission to obtain the object can be object bucket-level permission, object-level permission, or any other level. The type of permission to obtain the object can be public read, private, or other types. Of course, in addition to the object permission level and type mentioned above, the first server can also manage the object permission in a fine-grained manner through access control policies.

[0115] In a specific embodiment, since the first server has divided the target object into multiple chunks, the first server can select some chunks from the multiple chunks and send them to the first client. For example, assuming the first server has divided the object into four chunks, the first server can select the first and third chunks from the four chunks and send them to the first client. In a more specific embodiment, the chunks to be distributed to the first client can be selected based on a piece selection algorithm to maximize download speed, balance upload and download, and ensure file integrity. Alternatively, the chunks to be distributed to the first client can be selected based on a random selection algorithm, i.e., randomly selecting chunks from the multiple chunks and sending them to the client regardless of speed differences between clients. Alternatively, the chunks to be distributed to the first client can be selected based on a scarcity-first algorithm, i.e., prioritizing scarce chunks for distribution to the client. For example, if five clients already have chunk A, but only one client has chunk B, chunk B will be sent first. Alternatively, the chunks to be distributed to the first client can be selected based on a high-speed-first algorithm, i.e., prioritizing chunks provided by clients with faster download speeds. Of course, the above algorithms can also be combined to select a block from multiple blocks, so that a more scientific decision can be made. For example, the above algorithms can be combined in a weighted manner to make a decision.

[0116] Step S112: The first server sends the second part of the target object to the second client. Correspondingly, the second client receives the second part of the target object sent by the first server.

[0117] In a specific embodiment, in order to ensure security, the first server will send the second part of the object to the second client only when the second client has the permission to obtain the object. For the content about obtaining the object permission, please refer to the above and will not be elaborated here. The level of the first client's permission to obtain the object and the level of the second client's permission to obtain the object can be the same or different; the type of the first client's permission to obtain the object and the type of the second client's permission to obtain the object can be the same or different; the access control policy for the first client's permission to obtain the object and the access control policy for the second client's permission to obtain the object can be the same or different.

[0118] In a specific embodiment, the first server may select some blocks from a plurality of blocks and send them to the second client. For example, assuming that the first server divides the target object into 4 blocks, the first server may select the second block and the fourth block from the 4 blocks and send them to the first client. It is understandable that it is assumed here that completely different blocks are sent to the first client and the second client respectively. However, in actual applications, the first block and the third block may be sent to the first client, and the first block and the fourth block may be sent to the second client, or the first block and the third block may be sent to the first client, and the first block and the third block may be sent to the second client. In other words, the first part of the object and the second part of the target object may be completely identical, partially identical, or completely different, and no specific limitation is given here.

[0119] In a specific embodiment, the permission to obtain an object and the permission to obtain a seed file may be different. For example, since the confidentiality requirement of the object data is higher than that of the seed file data, the permission to obtain an object may be higher than the permission to obtain a seed file.

[0120] Step S113: The second client sends the second part of the target object to the first client. Correspondingly, the first client receives the second part of the target object sent by the second client.

[0121] In a specific embodiment, if a connection has been established between the first client and the second client before, and the connection has not been canceled, then the second client can directly send the second part of the object to the first client. If a connection has not been established between the first client and the second client, or the connection between the first client and the second client has been canceled, then the first client and the second client can establish or re-establish a connection. Since the second client has obtained a list of multi-source download providers, and the list of multi-source download providers includes the IP address and port number of the first client, etc. Therefore, the second client can locate the first client in the network through the IP address and port number of the first client and establish a network connection with the first client. After the connection between the first client and the second client is established, the first client can send the second part of the object to the first client.

[0122] In a specific embodiment, before the second client sends the second part of the object to the first client, token verification is required. Specifically, the first client first sends a first object acquisition request to the second client, wherein the first object acquisition request carries the first token. The second client receives the first object acquisition request, extracts the first token from the first object acquisition request, and sends the first token to the second server. After receiving the first token, the second server verifies the first token, and if the verification passes, sends a first verification pass message to the second client; if the verification fails, sends a first verification failure message to the second client. After receiving the first verification pass message, the second client sends the second part of the object to the first client; after receiving the first verification failure message, the second client refuses to send the second part of the object to the first client.

[0123] Step S114: The first client sends the first part of the target object to the second client. Correspondingly, the second client receives the first part of the target object sent by the first client.

[0124] In a specific embodiment, if a connection has previously been established between the first client and the second client and the connection has not been canceled, the first client can directly send the first part of the target object to the second client. If a connection has not been established between the first client and the second client, or the connection between the first client and the second client has been canceled, a connection can be established or re-established between the first client and the second client. Because the first client has obtained a list of multi-source download providers, which includes the second client's IP address and port number, etc., the first client can locate the second client on the network using the second client's IP address and port number and establish a network connection with the second client. After the connection between the first client and the second client is established, the second client can send the first part of the target object to the first client. In addition, after the second client completes sending the first part of the target object to the first client, the connection between the first client and the second client can be canceled to release network resources.

[0125] In a specific embodiment, before the first client sends the first part of the object to the second client, token verification is required. Specifically, the second client first sends a second object acquisition request to the first client, wherein the second object acquisition request carries the second token. The first client receives the second object acquisition request, extracts the second token from the second object acquisition request, and sends the second token to the second server. After receiving the second token, the second server verifies the second token, and if the verification passes, sends a second verification pass message to the first client; if the verification fails, sends a second verification failure message to the first client. After receiving the second verification pass message, the first client sends the first part of the target object to the second client; after receiving the second verification failure message, the first client refuses to send the first part of the target object to the second client.

[0126] In the above embodiment, it is assumed that the first client sends the first and third blocks of an object to the second client, and the second client sends the second and fourth blocks of the object to the first client. However, in actual applications, object storage systems often include more clients, such as a third client. Therefore, more complex situations may arise, such as the first client sending the first portion to the second client, the second client sending the first portion to the third client, and so on. Furthermore, the first client may send the first portion to the second client, and the third client may send the second portion to the second client. Furthermore, the first client may not send all of the multiple blocks downloaded from the first server to the second client, but may select only a portion to send to the second client. For example, the first client downloads the first and third blocks from the first server, but only sends the third block to the second client. Furthermore, in addition to sending blocks downloaded from the first server to the second client, the first client may also send blocks received from other clients to the second client. For example, the first client may download the first block from the first server and the third block from the third client, and then send the first and third blocks to the second client. The client can also obtain blocks in the target object only from other clients without obtaining blocks from the first server. For example, assuming that the first server divides the target object into 4 blocks, the first server can select the first block and the third block from the 4 blocks and send them to the first client, and select the second block and the fourth block from the 4 blocks and send them to the second client. The third client obtains the first block and the third block from the first client, and obtains the second block and the fourth block from the second client.

[0127] In the above embodiment, the execution order of the steps should be determined according to the logical relationship between the steps, and should not be understood as being limited to the execution order shown in Figure 3. For example, steps S105 and S106 may occur before step S102, step S103, or step S104, steps S105 and S106 may occur after step S107, steps S105, S106, and S108 may occur after step S109, steps S105, S106, S108, and S110 may occur after step S111, steps S105, S106, S108, S110, and S112 may occur after step S113, and so on.

[0128] Compared to conventional object storage systems, the relationship between the client and the first server in conventional object storage systems is relatively simple. Each client and the first server are connected via a single line, and each client only has an upload and download relationship with the first server. Therefore, the first server only needs to verify whether the client has download permissions. However, in the object storage system of the present application, the relationship between the first server and the client is much more complex. Not only does the first server and the client have data upload and download relationships, but a client can also theoretically have an upload and download relationship with any other client in the object storage system. In such a complex system, ensuring the security of objects, the first server, and the clients is crucial. Therefore, before the first server sends a seed file to the first and second clients, before the second server sends a list of multi-source download providers to the first and second clients, and before the first server sends object data to the first and second clients, it must first authenticate the identity of the first and second clients, confirm their permissions, and so on. Furthermore, before the first and second clients download data of a target object, they must also authenticate a token to ensure the security of the object download.

[0129] Because the target object's metadata and user data may be stored in different locations, suppose the target object's user data is stored in Beijing and its metadata is stored in Guangzhou. When a client in Shanghai needs to download the target object, it must first obtain the target object's metadata from the server in Guangzhou and then obtain the target object's user data from the server in Beijing. This results in significant latency. For example, suppose the latency for the Shanghai server to obtain the target object's metadata from the Guangzhou server is 10 seconds, and the latency for obtaining the target object's user data from the Beijing server is 1 second, resulting in a total latency of 11 seconds. However, if the torrent file downloaded by the client includes the target object's metadata, then when the client in Shanghai needs to download the target object, it only needs to obtain the target object's user data from the Beijing server. In this example, the total latency is 1 second, significantly reducing latency and improving the user experience. The above example assumes that the client, the metadata server, and the user data server are located in different regions. In actual applications, the client, the metadata server, and the user data server can all be located in the same region.

[0130] 5 , which is a schematic diagram of the structure of an object storage system provided by the present application. As shown in FIG5 , the object storage system of the present application includes: a client 210 , a first server 220 , and a second server 230 .

[0131] The client 210 includes a seed acquisition unit 211 , a list acquisition unit 212 , an object acquisition unit 213 , a receiving unit 214 , and a sending unit 215 .

[0132] The seed acquisition unit 211 is used to instruct the sending unit 215 to send a seed acquisition request to the first server 220, and receive the seed file sent by the first server 220 through the receiving unit 214. After obtaining the seed file, the seed acquisition unit 211 will notify the list acquisition unit 212 that it needs to obtain a list of multi-source download providers. The seed acquisition unit 211 can be used to provide the following functions: a seed file search function, where users can search and browse various seed files by keywords; download files from multiple sources simultaneously, thereby significantly improving download speed and efficiency; manage multiple download tasks, including setting download queues, priorities, and the number of tasks to be downloaded simultaneously, so as to better control the download order and speed; manage downloaded seed files, including operations such as adding, deleting, pausing, resuming, and re-downloading; support encrypted transmission and anonymous download functions to increase user privacy protection and security, etc.

[0133] The list acquisition unit 212 is used to instruct the sending unit 215 to send a multi-source download provider acquisition request to the second server 230, and receive the multi-source download provider list sent by the second server 230 through the receiving unit 214. After receiving the multi-source download provider acquisition request, the list acquisition unit 212 notifies the object acquisition unit 212 that the target object needs to be acquired.

[0134] The object acquisition unit 213 is configured to instruct the receiving unit 214 to receive a portion of the target object sent by the first server 220, a portion of the target object from other clients, or to receive the entire target object from one or more other clients. When the object acquisition unit 213 receives the target object from one or more other clients, it may receive a portion of the target object sent by client A, a portion of the target object sent by client B, and so on. The object acquisition unit 213 may have the following functions: uploading local files to the object storage service of the first server 220; combining various portions of the target object downloaded from different sources into an object; managing the uploaded target objects, including operations such as creation, deletion, copying, moving, and renaming; setting file access permissions, including public read / write and private read / write, to ensure file security and privacy; supporting multiple upload methods, including single file upload, batch upload, and resumable upload, to provide a more flexible and efficient upload experience; providing data encryption to encrypt uploaded objects for increased data security; and supporting data backup and recovery, allowing users to back up important data to the object storage service and restore it when needed.

[0135] The seed acquisition unit 211 may instruct the sending unit 215 to execute the steps performed by the first client in step S101 of the object downloading method shown in FIG4 , or instruct the sending unit 215 to execute the steps performed by the second client in step S105 of the object downloading method shown in FIG4 . The seed acquisition unit 211 may instruct the receiving unit 214 to execute the steps performed by the first client in step S104 of the object downloading method shown in FIG4 , or instruct the receiving unit 214 to execute the steps performed by the second client in step S106 of the object downloading method shown in FIG4 .

[0136] The list acquisition unit 212 may instruct the sending unit 215 to execute the steps performed by the first client in step S107 of the object downloading method shown in FIG4 , or instruct the sending unit 215 to execute the steps performed by the second client in step S108 of the object downloading method shown in FIG4 . The list acquisition unit 212 may instruct the receiving unit 214 to execute the steps performed by the first client in step S109 of the object downloading method shown in FIG4 , or instruct the receiving unit 214 to execute the steps performed by the second client in step S110 of the object downloading method shown in FIG4 .

[0137] The object acquisition unit 213 can instruct the receiving unit 214 to execute the steps performed by the first client in steps S111 and S113 of the object downloading method shown in Figure 4, or instruct the receiving unit 214 to execute the steps performed by the second client in steps S112 and S114 of the object downloading method shown in Figure 4.

[0138] It should be understood that the client in FIG5 is an exemplary division method, and the client 210 may also include more or fewer units. For example, the client 210 may also include a display unit for displaying seed files, lists of multi-source download providers, or objects. For another example, the seed acquisition unit 211, list acquisition unit 212, and object acquisition unit 213 may be combined into a single acquisition unit. For another example, the client 210 may not have a display unit. It should be understood that the above examples are for illustrative purposes only, and the specific units may be determined based on the actual application scenario, and this application does not impose any specific limitations.

[0139] In order to make this application more understandable, FIG5 only shows the internal structure of one client. In fact, the number of clients can be one or more. The internal structures of other clients are the same as the internal structure of the client 210 shown in FIG5 , and are not described in detail here.

[0140] The first server 220 includes a receiving unit 221 , a sending unit 222 , a seed generating unit 223 , and an object storing unit 224 .

[0141] The seed generating unit 223 is configured to receive a seed obtaining request sent by the client through the receiving unit 221 , generate a seed file for the object, and instruct the sending unit 222 to send the seed file to the second server for management.

[0142] Object storage unit 224 is used to instruct sending unit 222 to send the seed file to the second server and the object to the client. The object storage unit can include the following functions: dividing objects into multiple parts; storing large-scale data objects to support the storage and management of massive amounts of data; ensuring data security and reliability through data redundancy and backup mechanisms; providing high-availability storage services using a distributed architecture and data replication mechanism; offering multiple access interfaces and protocols, such as RESTful APIs and S3 APIs, allowing users to easily access and manage stored data; and providing data security and encryption. It can encrypt data storage to protect its confidentiality. It also provides access control and permission management to ensure that only authorized users can access and modify data; provides data management and retrieval functions, such as data classification, indexing, and searching; supports cross-regional data replication and backup; and verifies the integrity of downloaded data to ensure that the downloaded object is completely consistent with the original object.

[0143] The seed generation unit 223 can execute the steps executed by the first server in steps S101 and S105 of the object download method shown in Figure 4 through the receiving unit 221, execute step S102 of the object download method shown in Figure 4, and instruct the sending unit 222 to execute the steps executed by the first server in steps S103, S104, and S106 of the object download method shown in Figure 4.

[0144] The object storage unit 224 instructs the sending unit 222 to execute the steps executed by the first server in steps S111 and S112 of the object downloading method shown in FIG. 4 .

[0145] It should be understood that the first server 220 in Figure 5 is an exemplary division method. The first server 220 may also include more or fewer units. For example, the first server 220 may also include a display unit for displaying seed files or objects. For another example, the seed generation unit 223 and the object storage unit 224 may be combined into a single acquisition unit. For another example, the first server 220 may not have a display unit. It should be understood that the above examples are for illustrative purposes only. The specific units may be determined based on actual application scenarios and are not specifically limited in this application.

[0146] The seed generation unit 223 and the object storage unit 224 can both be implemented in software or hardware. For example, the implementation of the object storage unit 224 will be described below using the object storage unit 224 as an example. Similarly, the implementation of the seed generation unit 223 can refer to the implementation of the object storage unit 224.

[0147] As an example of a software functional unit, the object storage unit 224 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the object storage unit 224 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0148] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0149] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, the object storage unit 224 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0150] The multiple computing devices included in the object storage unit 224 can be distributed in the same region or in different regions. The multiple computing devices included in the object storage unit 224 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the object storage unit 224 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0151] It should be noted that, in other embodiments, the object storage unit 224 can be used to execute any step of the first server in the object download method shown in Figure 4, and the seed generation unit 223 can be used to execute any step of the first server in the object download method. The steps that the seed generation unit 223 and the object storage unit 224 are responsible for implementing can be specified as needed. The full functions of the first server can be realized by respectively implementing different steps in the object download method through the seed generation unit 223 and the object storage unit 224.

[0152] The second server 230 includes a receiving unit 231 , a sending unit 232 , and a list management unit 233 .

[0153] List management unit 233 receives seed files sent by first server 220 via receiving unit 231, receives requests from clients for obtaining multi-source download providers, and instructs sending unit 232 to send a list of multi-source download providers. List management unit 233 may also function as follows: storing a large number of seed files; coordinating data transfers between clients; recording the status and progress of each download task and providing this information to other clients; obtaining the IP addresses and port numbers of other clients to establish point-to-point data transfer connections; and coordinating data transfers between clients, scheduling them based on each client's upload and download speed to optimize the performance and efficiency of the entire download system.

[0154] The list management unit 233 can execute the steps executed by the second server in steps S103 and S107 of the object downloading method shown in Figure 4 through the receiving unit 231, and instruct the sending unit 232 to execute the steps executed by the second server in steps S109 and S110 of the object downloading method shown in Figure 4.

[0155] It should be understood that the second server 230 in FIG5 is an exemplary division method. The second server 230 may also include more or fewer units. For example, the second server 230 may also include a display unit for displaying a list of multi-source download providers. For another example, the receiving unit 231 and the sending unit 232 may be combined into a single acquisition unit. For another example, the second server 230 may not have a display unit. It should be understood that the above examples are for illustrative purposes only. The specific units may be determined based on the actual application scenario and are not specifically limited in this application.

[0156] The list management unit 233 can be implemented by software or hardware. For example, the object storage unit 224 can be implemented by software or hardware.

[0157] As an example of a software functional unit, the list management unit 233 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the list management unit 233 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0158] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0159] As an example of a hardware functional unit, the list management unit 233 may include at least one computing device, such as a server. Alternatively, the list management unit 233 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0160] The multiple computing devices included in the list management unit 233 can be distributed in the same region or in different regions. The multiple computing devices included in the list management unit 233 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the list management unit 233 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0161] It should be noted that, in other embodiments, the list management unit 233 may be used to execute any steps of the second server in the object downloading method shown in FIG. 4 , and the steps that the list management unit 233 is responsible for implementing may be specified as needed.

[0162] This application also provides a computing device 400. As shown in Figure 6, computing device 400 includes a bus 402, a processor 404, a memory 406, and a communication interface 408. Processor 404, memory 406, and communication interface 408 communicate with each other via bus 402. Computing device 400 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 400.

[0163] Bus 402 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG6 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 402 may include a path for transmitting information between various components of computing device 400 (e.g., memory 406, processor 404, and communication interface 408).

[0164] The processor 404 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0165] The memory 406 may include volatile memory, such as random access memory (RAM). The processor 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0166] When the computing device is a client, the memory 406 stores executable program code, and the processor 404 executes this executable program code to respectively implement the functions of the aforementioned seed acquisition unit, list acquisition unit, and object acquisition unit, thereby implementing the steps performed by the client in the object download method. In other words, the memory 406 stores instructions for executing the steps performed by the client in the object download method. Alternatively, the memory 406 stores executable code, and the processor 404 executes this executable code to respectively implement the aforementioned client functions, thereby implementing the steps performed by the client in the object download method. In other words, the memory 406 stores instructions for executing the steps performed by the client in the object download method.

[0167] When the computing device is a first server, memory 406 stores executable program code, and processor 404 executes this executable program code to implement the functions of the aforementioned seed generation unit and object storage unit, respectively, thereby implementing the steps performed by the first server in the object download method. That is, memory 406 stores instructions for executing the steps performed by the first server in the object download method. Alternatively, memory 406 stores executable code, and processor 404 executes this executable code to implement the functions of the aforementioned first server, respectively, thereby implementing the steps performed by the first server in the object download method. That is, memory 406 stores instructions for executing the steps performed by the first server in the object download method.

[0168] When the computing device is a second server, memory 406 stores executable program code, and processor 404 executes this executable program code to implement the functions of the aforementioned list management unit, thereby implementing the steps performed by the second server in the object download method. In other words, memory 406 stores instructions for executing the steps performed by the second server in the object download method. Alternatively, memory 406 stores executable code, and processor 404 executes this executable code to implement the functions of the aforementioned second server, thereby implementing the steps performed by the second server in the object download method. In other words, memory 406 stores instructions for executing the steps performed by the second server in the object download method.

[0169] The communication interface 408 uses a receiving unit and a transmitting unit such as, but not limited to, a network interface card and a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0170] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0171] As shown in Figure 7 , the computing device cluster includes at least one computing device 400. The memory 406 in one or more computing devices 400 in the computing device cluster may store the same instructions for executing the steps executed by the first server and the second server in the object downloading method described in Figure 4 .

[0172] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the steps performed by either the first server or the second server in the object downloading method. In other words, the combination of one or more computing devices 400 can collectively execute instructions for executing the steps performed by either the first server or the second server in the object downloading method.

[0173] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions, each for executing a portion of the functions of the first server or the second server. In other words, the instructions stored in the memory 406 in different computing devices 400 can implement the functions of one or more of the seed generation unit 223 and the object storage unit 224, or the functions of the list management unit.

[0174] In some possible implementations, taking the first server as an example, one or more computing devices in the computing device cluster can be connected via a network. The network can be a wide area network or a local area network, etc. Figure 8 shows a possible implementation. As shown in Figure 8, two computing devices 400A and 400B are connected via a network. Specifically, the connection to the network is made through the communication interface in each computing device. In this type of possible implementation, the memory 406 in the computing device 400A stores instructions for executing the functions of the seed generation unit 223 in the first server. At the same time, the memory 406 in the computing device 400B stores instructions for executing the functions of the object storage unit 224.

[0175] The connection method between the computing device clusters shown in Figure 8 can be considered to take into account the needs of the object download method provided in this application (such as storing a large amount of data), so it is considered to hand over the functions implemented by the object storage unit 224 to the computing device 400B for execution.

[0176] It should be understood that the functionality of the computing device 400A shown in FIG8 may also be accomplished by multiple computing devices 400. Similarly, the functionality of the computing device 400B may also be accomplished by multiple computing devices 400.

[0177] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection method of the computing device cluster described in Figures 7 and 8. However, the memory 406 in one or more computing devices 400 in this computing device cluster can store the same instructions for executing the steps performed by the first server in the object download method.

[0178] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store a portion of the instructions for executing the first server in the object downloading method. In other words, the combination of one or more computing devices 400 can collectively execute the instructions for executing the steps performed by the first server in the object downloading method.

[0179] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions for respectively executing the functions of the first server and the second server. In other words, the instructions stored in the memory 406 in different computing devices 400 can implement the functions of one or more of the first server and the second server.

[0180] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be executed on a computing device or stored in any available medium. When the computer program product is executed on at least one computing device, the computer program product causes the at least one computing device to execute one or more of the steps performed by the client, the first server, or the second server in the object download method shown in FIG. 4 .

[0181] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute one or more steps performed by the client, the first server, or the second server in the object download method.

[0182] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

[0183] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims

1. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first client sends a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; The first server sends the seed file to the first client based on the first seed acquisition request; The first client sends a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server sends a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The first client receives a first part of the target object sent by the first server, and receives a second part of the target object sent by the second client.

2. The method according to claim 1, characterized in that The method further comprises: When the first client is authenticated successfully, the second server sends the first token to the first client; Before receiving the second part of the target object sent by the second client, the method further includes: The first client sends the first token to the second client; The second client sends the first token to a second server; The second server verifies the first token, and if the verification is successful, sends a first verification success message to the second client; After receiving the first verification success message, the second client sends the second part of the target object to the first client.

3. The method according to claim 2, characterized in that Before sending the second part of the target object to the first client, the method further includes: The second client sends a second seed acquisition request to the first server, wherein the second seed acquisition request is used to request a seed file of the target object; The second client runs the seed file and sends a second multi-source download provider acquisition request to the second server, wherein the second multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server authenticates the second client based on the second multi-source download provider acquisition request, and sends a second multi-source download provider list to the second client if the second client passes the authentication; The second client receives a second part of the target object sent by the first server.

4. The method according to claim 3, characterized in that The method further comprises: When the second client is authenticated successfully, the second server sends a second token to the second client; Before receiving the first part of the target object sent by the first client, the method further includes: The second client sends the second token to the first client; The first client sends the second token to the second server; The second server verifies the second token, and if the verification is successful, sends a second verification success message to the first client; After receiving the second verification success message, the first client sends the first part of the target object to the second client.

5. The method according to any one of claims 1 to 4, characterized in that: The first server is used to open a first permission for obtaining the seed file and a second permission for obtaining the target object to the first client, wherein the first permission is lower than the second permission.

6. The method according to any one of claims 1 to 5, characterized in that: The first server includes a metadata server and a user data server, and the target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server.

7. The method according to claim 6, characterized in that The seed file includes metadata of the target object.

8. The method according to any one of claims 1 to 7, characterized in that: The first server sends the seed file to the first client based on the seed acquisition request, including: The first server creates the seed file for the target object based on the seed acquisition request, and sends the seed file to the first client.

9. The method according to any one of claims 1 to 8, characterized in that: The second server sends the first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, including: The second server authenticates the first client based on the first multi-source download provider acquisition request, and sends a first multi-source download provider list to the first client if the first client passes the authentication.

10. An object downloading method, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first client sends a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; Receiving the seed file sent by the first server based on the first seed acquisition request; Sending a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the multi-source download of the target object for the first client; Receiving a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; A first part of the target object sent by the first server is received, and a second part of the target object sent by the second client is received.

11. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first server receives a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object; receiving a second seed acquisition request sent by the second client, wherein the second seed acquisition request is used to request a seed file of the target object; Sending the seed file to the first client and the second client respectively; Sending a first portion of the target object to the first client; A second portion of the target object is sent to the second client.

12. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The second server receives the seed file of the target object sent by the first server, and establishes a multi-source download provider list based on the seed file; Receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with downloads of the target object; Based on the first multi-source download provider acquisition request, a first multi-source download provider list is sent to the first client, the first multi-source download provider list includes the second client, and the first multi-source download provider list belongs to the multi-source download provider list.

13. An object storage system, characterized in that The object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, The first client is used to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; The first server is used for sending the seed file to the first client based on the first seed acquisition request; The first client is used to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The first client is used to receive the first part of the target object sent by the first server, and receive the second part of the target object sent by the second client.

14. The system according to claim 13, characterized in that The second server is further configured to send a first token to the first client when the first client is authenticated successfully; The first client is further used to send the first token to the second client; The second client is further used to send the first token to a second server; The second server is further used to verify the first token, and if the verification is successful, send a first verification success message to the second client; The second client is further configured to send the second part of the target object to the first client after receiving the first verification success message.

15. The system according to claim 14, characterized in that The second client is used to send a second seed acquisition request to the first server, wherein the second seed acquisition request is used to request a seed file of the target object; The second client is used to run the seed file and send a second multi-source download provider acquisition request to the second server, wherein the second multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server is used to authenticate the second client based on the second multi-source download provider acquisition request, and send a second multi-source download provider list to the second client if the second client passes the authentication, wherein the second multi-source download provider list includes the first client; The second client is used to receive the second part of the target object sent by the first server.

16. The system according to claim 15, characterized in that The second server is further configured to send a second token to the second client if the second client is authenticated successfully; The second client is further used to send the second token to the first client; The first client is further used to send the second token to the second server; The second server is further used to verify the second token, and if the verification is successful, send a second verification success message to the first client; The first client is further configured to send the first part of the target object to the second client after receiving the second verification success message.

17. The system according to any one of claims 13 to 16, characterized in that: The first server is used to open a first permission for obtaining the seed file and a second permission for obtaining the target object to the first client, wherein the first permission is lower than the second permission.

18. The system according to any one of claims 13 to 17, characterized in that: The first server includes a metadata server and a user data server, and the target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server.

19. The system according to claim 18, characterized in that The seed file includes metadata of the target object.

20. The system according to any one of claims 13 to 19, characterized in that: The first server is used to create the seed file for the target object based on the seed acquisition request, and send the seed file to the first client.

21. The system according to any one of claims 13 to 20, characterized in that: The second server is used for authenticating the first client based on the first multi-source download provider acquisition request, and sending the first multi-source download provider list to the first client if the first client is authenticated successfully.

22. A first client, wherein: The first client belongs to an object storage system, the object storage system further includes a second client, a first server and a second server, cloud computing includes a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first client includes: A sending unit, configured to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; A receiving unit, configured to receive the seed file sent by the first server based on the first seed acquisition request; The sending unit is used to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the multi-source download of the target object for the first client; The receiving unit is used to receive a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The receiving unit is used to receive a first part of the target object sent by the first server, and receive a second part of the target object sent by the second client.

23. A first server, characterized in that: The first server belongs to an object storage system, the object storage system further includes a first client, a second client and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first server includes: A receiving unit, configured to receive a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object; The receiving unit is further configured to receive a second seed acquisition request sent by a second client, wherein the second seed acquisition request is used to request a seed file of the target object; A sending unit, configured to send the seed file to the first client and the second client respectively; The sending unit is further configured to send the first part of the target object to the first client; The sending unit is further configured to send the second part of the target object to the second client.

24. A second server, characterized in that: The second server belongs to an object storage system, the object storage system further includes a first client, a second client and a first server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the second server includes: A receiving unit, configured to receive a seed file of a target object sent by the first server, and to establish a multi-source download provider list based on the seed file; The receiving unit is further configured to receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with downloads of the target object; The sending unit is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client and the first multi-source download provider list belongs to the multi-source download provider list.

25. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

26. A computing cluster, characterized in that: The computing cluster includes a plurality of computer devices, each of which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to claims 1-9 when executing the computer program.

27. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to claims 1 to 9.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed on a computer device, enable the computer device to execute the method according to any one of claims 1 to 9.

Images