Object downloading method, device and system, cluster and medium

Abstract

The invention provides an object downloading method, device and system, a cluster and a medium. The method comprises the following steps: a first client sends a first seed acquisition request to an object storage service on a first server, wherein the first seed acquisition request is used for requesting a seed file of a target object; the first server sends the seed file to the first client based on the first seed acquisition request; the first client sends a first multi-source downloading provider acquisition request to a second server based on the seed file, wherein the first multi-source downloading provider acquisition request is used for requesting to provide a list of at least one client for downloading the target object; the second server sends a first multi-source downloading provider list to the first client based on the first multi-source downloading provider acquisition request; and the first client receives the first part in the target object sent by the first server and receives the second part in the target object sent by the second client, so that the downloading speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of storage, and in particular to an object downloading method, device, system, cluster and medium. Background Art

[0002] With the rapid development of the Internet, object storage has been widely used in various Internet scenarios, providing massive, secure, low-cost, and highly reliable unstructured storage services. However, the current object storage can only access the same object by initiating multiple requests for the same object. This may cause great pressure on the object storage server. Once the object storage server fails, it will lead to the inability to download the object, or even cause the loss of the object, or the service provider needs to pay a very high cost to ensure the user's download experience. Therefore, it is impossible to provide high-performance, low-cost support in scenarios such as live broadcast on demand, concurrent image construction of cloud services, and AI training. Summary of the invention

[0003] The present application provides an object downloading method, device, system, cluster and medium, which can download target objects from multiple download sources, thereby improving the downloading speed.

[0004] In a first aspect, an object download method is provided. The method is applied to an object storage system. The object storage system includes a first client, a second client, a first server, and a second server. The first server deploys an object storage service, and the first server and the second server are the same server or different servers.

[0005] The first client sends a first seed acquisition request to the object storage service on the first server. Correspondingly, the first server receives the first seed acquisition request sent by the first client. The first seed acquisition request is used to request a seed file of a target object. The target object is an object stored in the first server. The first server sends the seed file to the first client based on the first seed acquisition request. Correspondingly, the first client receives the seed file sent by the first server based on the first seed acquisition request.

[0006] The second client sends a second seed acquisition request to the object storage service on the first server. Correspondingly, the first server receives the second seed acquisition request sent by the second client. The second seed acquisition request is used to request a seed file of a target object. The first server sends the seed file to the second client based on the second seed acquisition request. Correspondingly, the second client receives the seed file sent by the first server based on the second seed acquisition request.

[0007] The first client sends a first multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server receives the first multi-source download provider acquisition request sent by the first client based on the seed file. The second server sends a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request. Correspondingly, the first client receives the first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request. The first multi-source download provider acquisition request is used to request a list of at least one client that downloads the target object. For example, it may include the second client.

[0008] The second client sends a second multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server receives the second multi-source download provider acquisition request sent by the second client based on the seed file. The second server sends a second multi-source download provider list to the second client based on the second multi-source download provider acquisition request. Accordingly, the second client receives the second multi-source download provider list sent by the second server based on the second multi-source download provider acquisition request. The second multi-source download provider acquisition request is used to request a list of at least one client that downloads the target object. For example, it may include the first client.

[0009] The first client receives a first part of the target object sent by the first server, and the second client receives a second part of the target object sent by the first server.

[0010] The first client sends a first download request to the second client based on the first multi-source download provider list. Correspondingly, the second client receives the first download request sent by the first client based on the first multi-source download provider list. The second client sends the second part of the target object to the first client. Correspondingly, the first client receives the second part of the target object sent by the second client.

[0011] In the above scheme, the object is divided into multiple parts. After the second client downloads the second part of the object, the first client can download the first part of the object from the first server and download the second part of the object from the second client. Therefore, downloading objects from multiple channels is realized, which improves the speed of object downloading. Moreover, even if the first server fails, the object can still be downloaded from other clients, reducing the risk of being unable to download the object due to a failure of the first server. At this time, there is no need to perform active-active backup of the first server, thereby reducing costs.

[0012] In some possible designs, in order to ensure data security, before the first client sends the first part of the target object to the second client, it is also necessary to authenticate the second client. Similarly, before the second client sends the second part of the target object to the first client, it is also necessary to authenticate the first client. In the case that the first client is authenticated successfully, the second server sends the first token to the first client. In the case that the second client is authenticated successfully, the second server sends the second token to the second client.

[0013] The first client sends the first token to the second client. Accordingly, the second client receives the first token sent by the first client. The second client sends the first token to the second server. Accordingly, the second server receives the first token sent by the second client. The second server verifies the first token and, if the verification is successful, sends a first verification pass message to the second client. Accordingly, the second client receives the first verification pass message sent by the second server. The second client sends the second part of the target object to the first client only after receiving the first verification pass message.

[0014] The second client sends the second token to the first client. Accordingly, the first client receives the second token sent by the second client. The first client sends the second token to the second server. Accordingly, the second server receives the second token sent by the first client. The second server verifies the second token and, if the verification is successful, sends a second verification pass message to the second client. Accordingly, the second client receives the second verification pass message sent by the second server. The second client sends the first part of the target object to the first client only after receiving the second verification pass message.

[0015] In the above scheme, because the relationship between the client and the first server is one-to-one in the previous object storage system, the relationship is relatively simple and does not require complex verification. However, in this application, the relationship between the members within the system is very complex. In addition to downloading data from the first server, the client can also download data from other clients. Therefore, it is necessary to verify the identity of the other party before sending data.

[0016] In some possible designs, the first server is used to open a first permission to obtain the seed file and a second permission to obtain the target object to the first client. Similarly, the first server is used to open a first permission to obtain the seed file and a second permission to obtain the target object to the second client, wherein the first permission is lower than the second permission.

[0017] In the above solution, because the confidentiality level of the data contained in the seed file is not as high as that of the target object, the first permission for obtaining the seed may be lower than the second permission for obtaining the target object.

[0018] In some possible designs, the first server includes a metadata server and a user data server. The target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server. At this time, after the first server receives the first seed acquisition request, it obtains the metadata of the target object from the metadata server. Then, the first server generates a seed file for the target object based on the metadata of the target object. Therefore, the generated seed file includes the metadata of the target object. Therefore, when the second client sends a second seed acquisition request to the first server, the seed file sent by the first server to the second client includes the metadata of the target object. The second client only needs to download the user data of the target object, so the download delay can be effectively reduced. Here, the first server and the second server can be different servers in the same area, or different servers in different areas.

[0019] In some possible designs, the first server creates the seed file for the target object based on the seed acquisition request, and sends the seed file to the first client.

[0020] In some possible designs, the second server authenticates the first client based on the first multi-source download provider acquisition request, and sends the first multi-source download provider list to the first client if the first client is authenticated. The second server authenticates the second client based on the second multi-source download provider acquisition request, and sends the second multi-source download provider list to the second client if the second client is authenticated. Here, the first multi-source download provider list and the second multi-source download provider list may be the same or different.

[0021] In a second aspect, an object storage system is provided, the object storage system comprising a first client, a second client, a first server and a second server. The first server deploys an object storage service, and the first server and the second server are the same server or different servers.

[0022] The first client is used to send a first seed acquisition request to the object storage service on the first server. Accordingly, the first server is used to receive the first seed acquisition request sent by the first client. The first seed acquisition request is used to request a seed file of a target object. The target object is an object stored in the first server. The first server is used to send the seed file to the first client based on the first seed acquisition request. Accordingly, the first client is used to receive the seed file sent by the first server based on the first seed acquisition request.

[0023] The second client is used to send a second seed acquisition request to the object storage service on the first server. Accordingly, the first server is used to receive the second seed acquisition request sent by the second client. The second seed acquisition request is used to request a seed file of a target object. The first server is used to send the seed file to the second client based on the second seed acquisition request. Accordingly, the second client is used to receive the seed file sent by the first server based on the second seed acquisition request.

[0024] The first client is used to send a first multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server is used to receive the first multi-source download provider acquisition request sent by the first client based on the seed file. The second server is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request. Accordingly, the first client is used to receive the first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request. The first multi-source download provider acquisition request is used to request a list of at least one client that downloads the target object. For example, it may include a second client.

[0025] The second client is used to send a second multi-source download provider acquisition request to the second server based on the seed file. Accordingly, the second server is used to receive the second multi-source download provider acquisition request sent by the second client based on the seed file. The second server is used to send a second multi-source download provider list to the second client based on the second multi-source download provider acquisition request. Accordingly, the second client is used to receive a second multi-source download provider list sent by the second server based on the second multi-source download provider acquisition request. The second multi-source download provider acquisition request is used to request a list of at least one client that downloads the target object. For example, it may include the first client.

[0026] The first client is used to receive a first part of the target object sent by the first server, and the second client is used to receive a second part of the target object sent by the first server.

[0027] The first client is used to send a first download request to the second client based on the first multi-source download provider list. Correspondingly, the second client is used to receive the first download request sent by the first client based on the first multi-source download provider list. The second client is used to send the second part of the target object to the first client. Correspondingly, the first client is used to receive the second part of the target object sent by the second client.

[0028] In some possible designs, the first client is used to authenticate the second client before sending the first part of the target object to the second client. Similarly, the second client is used to authenticate the first client before sending the second part of the target object to the first client. The second server is used to send the first token to the first client if the first client is authenticated successfully; and send the second token to the second client if the second client is authenticated successfully.

[0029] The first client is used to send the first token to the second client. Accordingly, the second client is used to receive the first token sent by the first client. The second client is used to send the first token to a second server. Accordingly, the second server is used to receive the first token sent by the second client. The second server is used to verify the first token, and if the verification is successful, send a first verification pass message to the second client. Accordingly, the second client is used to receive the first verification pass message sent by the second server. After receiving the first verification pass message, the second client is used to send the second part of the target object to the first client.

[0030] The second client is used to send the second token to the first client. Accordingly, the first client is used to receive the second token sent by the second client. The first client is used to send the second token to the second server. Accordingly, the second server is used to receive the second token sent by the first client. The second server is used to verify the second token and, if the verification is successful, send a second verification pass message to the second client. Accordingly, the second client is used to receive the second verification pass message sent by the second server. After receiving the second verification pass message, the second client is used to send the first part of the target object to the first client.

[0031] In some possible designs, the first server is used to open a first permission to obtain the seed file and a second permission to obtain the target object to the first client. Similarly, the first server is used to open a first permission to obtain the seed file and a second permission to obtain the target object to the second client, wherein the first permission is lower than the second permission.

[0032] In some possible designs, the first server includes a metadata server and a user data server. The target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server. At this time, after receiving the first seed acquisition request, the first server obtains the metadata of the target object from the metadata server, and then generates a seed file for the target object based on the metadata of the target object. Therefore, the generated seed file includes the metadata of the target object. Therefore, when the second client sends a second seed acquisition request to the first server, the seed file sent by the first server to the second client includes the metadata of the target object. The second client only needs to download the user data of the target object, so the download delay can be effectively reduced. Here, the first server and the second server can be different servers in the same area, or different servers in different areas.

[0033] In some possible designs, the first server is used to create the seed file for the target object based on the seed acquisition request, and send the seed file to the first client.

[0034] In some possible designs, the second server is used to authenticate the first client based on the first multi-source download provider acquisition request, and send the first multi-source download provider list to the first client if the first client is authenticated. The second server is used to authenticate the second client based on the second multi-source download provider acquisition request, and send the second multi-source download provider list to the second client if the second client is authenticated. Here, the first multi-source download provider list and the second multi-source download provider list may be the same or different.

[0035] In a third aspect, a first client is provided, wherein the first client belongs to an object storage system, the object storage system further includes a second client, a first server, and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first client includes:

[0036] A sending unit, configured to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object;

[0037] A receiving unit, configured to receive the seed file sent by the first server based on the first seed acquisition request;

[0038] The sending unit is used to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the multi-source download of the target object for the first client;

[0039] The receiving unit is used to receive a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client;

[0040] The receiving unit is used to receive a first part of the target object sent by the first server, and receive a second part of the target object sent by the second client.

[0041] In a fourth aspect, a first server is provided, the first server belongs to an object storage system, the object storage system further includes a first client, a second client, and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first server includes:

[0042] A receiving unit, configured to receive a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object;

[0043] The receiving unit is further configured to receive a second seed acquisition request sent by a second client, wherein the second seed acquisition request is used to request a seed file of the target object;

[0044] A sending unit, configured to send the seed file to the first client and the second client respectively;

[0045] The sending unit is further configured to send the first part of the target object to the first client;

[0046] The sending unit is further configured to send the second part of the target object to the second client.

[0047] In a fifth aspect, a second server is provided, the second server belongs to an object storage system, the object storage system further includes a first client, a second client, and a first server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the second server includes:

[0048] A receiving unit, configured to receive a seed file of a target object sent by the first server, and to establish a multi-source download provider list based on the seed file;

[0049] The receiving unit is further configured to receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with downloads of the target object;

[0050] The sending unit is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client and the first multi-source download provider list belongs to the multi-source download provider list.

[0051] In a sixth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps performed according to any one of the first client, the second client, the first server, and the second server in the first aspect are implemented.

[0052] In the seventh aspect, a computing cluster is provided, which includes multiple computer devices, each of which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it implements the steps performed by any angle of the first server or the second server in the first aspect.

[0053] In an eighth aspect, a computer program product comprising instructions is provided, and when the instructions are executed by a computing device cluster, the computing device cluster executes the steps executed from any perspective of the first server or the second server in the first aspect.

[0054] In a ninth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer device, the computer device executes the steps performed by any one of the client, the first server or the second server in the first aspect of the claim. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the background technology, the drawings required for use in the embodiments of the present invention or the background technology will be described below.

[0056] Figure 1 A schematic diagram of the structure of an object storage provided for this application;

[0057] Figure 2 A schematic diagram of the structure of an object storage system provided for this application;

[0058] Figure 3 A schematic diagram of an object storage server provided for this application;

[0059] Figure 4 It is a flowchart of an object downloading method provided by this application;

[0060] Figure 5 It is a structural diagram of an object storage system provided by this application;

[0061] Figure 6 It is a structural schematic diagram of a computing device provided by the present application;

[0062] Figure 7 It is a schematic diagram of the structure of a computing cluster provided by this application;

[0063] Figure 8 It is a schematic diagram of the structure of another computing cluster provided in this application. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0065] Objects mainly refer to unstructured data such as movies, office documents, and images.

[0066] Object storage service (OBS) is a way to store unstructured data such as movies, office documents, and images. Object storage provides a scalable, self-managed, protected, and shareable storage solution. Object storage uses a flat address space to store data, which does not have a hierarchy of directories and files. Thanks to this, an object storage system can store a large number of objects. Figure 1 As shown in Figure 1, an object can include user data, related metadata (such as size, date, owner, etc.), and other attributes of the data (such as access characteristics, etc.). Each object is identified by a unique ID, which is called an object ID. Object IDs are generated using a dedicated algorithm (such as a hash value of data) to ensure that each object ID is unique.

[0067] Object-based storage devices (OSDs) are devices that store objects. Object-based storage devices usually use low-cost and high-density storage devices (such as hard disks) to store objects. When more capacity is needed, more hard disks can be added.

[0068] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an object storage system provided by this application. Figure 2 As shown, the object storage system of this embodiment includes: multiple clients 110 , a first server 120 and a second server 130 .

[0069] The client 110 is a software tool or library for interacting with the server, and provides an application programming interface (API) or a command line tool to enable users to perform basic operations such as uploading, downloading, deleting, copying, renaming, and managing buckets and data stored in OSDs. The first client may be an OBS client, and in order to implement multi-source downloading of objects, functions related to multi-source downloading may be newly added on the basis of the original functions of the OBS client, such as functions of obtaining seed files, running seed files, etc. Among them, multi-source downloading means that in addition to being able to download objects through the server, objects can also be downloaded through other clients.

[0070] The first server 120 can run the operating system of the OSD and provide a server for data storage, acquisition, and management services. Figure 3 As shown, the first server 120 provides two main services as an object storage server: metadata service and storage service. When receiving the unstructured data to be stored, the object storage server divides the unstructured data into two parts: user data and metadata. Then, a dedicated algorithm is applied to generate an object ID, which generates a unique object ID based on the content of the user data. In order to facilitate subsequent access, the object storage server uses the metadata service to store metadata and object ID, and uses the storage service to store the user data (object) in the object storage device. After the storage is successful, the stored content can be obtained through the object ID. Because object storage is usually a cloud service, in order to enable object storage to automatically adjust its resource scale as needed to meet the needs of different workloads and provide sufficient flexibility and scalability to cope with changing needs and traffic peaks, user data and metadata are often stored separately, and even the storage locations of user data and metadata can be far apart, for example, user data is stored in Beijing, metadata is stored in Guangzhou, and so on. In order to realize multi-source downloading of objects, the first server 120 can add new functions related to multi-source downloading based on the functions of the OBS server, such as generating seed files of the objects, publishing the seed files to the second server 130, opening the client 110 to download the seed files, downloading objects in a multi-source download manner (described in detail below), etc.

[0071] The second server 130 is usually used to manage multi-source download providers. Specifically, the second server 130 is used to register the information of the multi-source download providers. When the client 110 joins the object storage system, it will register its own information with the second server 130, including the Internet Protocol (IP) address, port number, etc. between the networks; maintain the list of multi-source download providers, that is, the second server 130 maintains the information of all participants who participate in the multi-source download of the object; provide the list of multi-source download providers: that is, when the client 110 wants to download a specific file, it can request the list of multi-source download providers from the second server 130. The second server 130 will return the information of other clients 110 for the object in the object storage system to help the client 110 (i.e., the downloader) establish a connection and start downloading; track the download progress, that is, the second server 130 will track the download progress of the downloader and notify other clients 110 of it to help them select a suitable client 110 for downloading. The second server 130 can be a tracker server with the function of processing objects added.

[0072] Figure 2 The object storage system shown is only illustrated by taking two clients 110, a first server 120 and a second server 130 as an example. In actual applications, there can be more than three clients 110, more than two first servers 120, more than two second servers 130, and the first server and the second server can be merged into the same server.

[0073] In addition, the client, the first server, and the second server may be located in the same cloud, the client and the first server may be located in the same cloud, and the second server may be located in another cloud, the client may be located in one cloud, and the first server and the second server may be located in another cloud, or the client, the first server, and the second server may be located in different clouds respectively.

[0074] See also Figure 4 , Figure 4 This is a flowchart of an object download method provided by this application. Figure 4 As shown, the object download method provided by this application includes the following steps:

[0075] Step S101: a first client sends a first seed acquisition request to a first server. Correspondingly, the first server receives the first seed acquisition request sent by the first client.

[0076] In a specific embodiment, the first client may be Figure 2 In the embodiment shown, the client 110, the first server may be Figure 2 For details, see the first server 120 in the embodiment shown. Figure 2 The description of the related embodiments will not be elaborated here.

[0077] In a specific embodiment, the first seed acquisition request is used to request a seed file of a target object, which is one or more objects in the first server.

[0078] Step S102: the first server generates a seed file for the target object based on the first seed acquisition request.

[0079] In a specific embodiment, the seed file contains metadata of one or more files such as object ID, object size, object hash value, uniform resource locator (URL) of the second server, etc. Among them, the object ID is the unique identifier of the target object that the first client is going to download. The object size is the size of the data volume of the target object, for example, the size of the object is 1G bit, etc. The object hash value is the value obtained after performing a hash operation on the target object, which is used to verify the integrity of the downloaded object and ensure that the downloaded target object is consistent with the original object. The URL of the second server is used to identify and locate the second server, and the first client can access the second server according to the URL of the second server. Optionally, the seed file also includes metadata of the object.

[0080] In a specific embodiment, in addition to generating a seed file, the first server can also divide the target object into multiple parts, each of which can be called a block. When the target object is divided, the divided blocks can be blocks of equal size or blocks of unequal sizes, which are not specifically limited here. When the target object is divided into blocks of unequal sizes, the large blocks can be sent to clients with more ample bandwidth, and the small blocks can be sent to clients with less bandwidth, thereby improving the overall download effect of the entire network. In addition, the way different objects are divided can also be different. For example, object A can be divided into multiple blocks in units of 256 kilobits, object B can be divided into multiple blocks in units of 128 kilobits, and so on.

[0081] Step S103: The first server sends a seed file to the second server. Correspondingly, the second server receives the seed file sent by the first server.

[0082] In a specific embodiment, after receiving the seed file sent by the first server, the second server will establish a multi-source download provider list for the object in the first server. The multi-source download provider list may include the object ID, the IP address of the client, etc. In addition, the multi-source download provider list may also include one or more of the client's port number, the client's upload speed, the client's download speed, the connection status (e.g., connected, disconnected, connecting), the object size, the object hash value, etc. The multi-source download provider list may be changeable, that is, when a client leaves the object storage system, the client may be deleted from the multi-source download provider list, and when a client joins the object storage system, the client may be added to the multi-source download provider list. In addition, in order to save resources, when the multi-source download provider list has not been used for a long time, the second server may delete the multi-source download provider list.

[0083] Step S104: The first server sends a seed file to the first client. Correspondingly, the first client receives the seed file sent by the first server.

[0084] In a specific embodiment, the first server sends the seed file to the first client only when the first client is authenticated. The first server authenticates the first client to ensure that only trusted clients can access, use or interact with the first server, prevent unauthorized clients from accessing the first server, and protect the security and confidentiality of the seed file and the object. The authentication method may include one or more of identification authentication, certificate authentication, and token authentication. Identification authentication may be authentication of the serial number, media access control address (MAC) address, international mobile equipment identity (IMEI) number, etc. of the client, thereby preventing the use of forged or invalid clients for access. Certificate authentication may be authentication of one or more of the public key of the client, the identity information of the client, and the digital signature, etc., thereby ensuring the legitimacy of the client through the validity and integrity of the certificate. Token authentication is authentication of one or more of a specific key, access token, or client token, etc., thereby confirming the identity and authority of the client.

[0085] In a specific embodiment, in order to ensure the security of object storage, the first server will send the seed file to the first client only when the first client has the permission to obtain the seed file. The level of permission to obtain the seed file can be any one of the permissions at the object bucket level, the object level, etc. The type of permission to obtain the seed file can be public read, private, etc. Of course, in addition to the level of permission to obtain the seed file and the type of permission mentioned above, the first server can also manage the permission to obtain the seed file in a fine-grained manner through the access control policy. The access control policy is a permission definition method based on the policy language, and more complex permission rules can be defined as needed. Users can freely choose the setting method, level and type of permissions as needed.

[0086] Step S105: The second client sends a second seed acquisition request to the first server. Correspondingly, the first server receives the second seed acquisition request sent by the second client.

[0087] In a specific embodiment, the second client may be Figure 2 For details, see the client 110 in the embodiment shown. Figure 2 The description of the related embodiments will not be elaborated here.

[0088] In a specific embodiment, the second seed acquisition request is used to request a seed file of the target object.

[0089] Step S106: The first server sends a seed file to the second client. Correspondingly, the second client receives the seed file sent by the first server.

[0090] In a specific embodiment, because the first server has already generated a seed file when the first client sends a first seed acquisition request to the first server, when the second client sends a second seed acquisition request to the first server, the first server determines that the seed file has been generated for the target object. Therefore, the first server can directly send the generated seed file to the second client without generating the seed file again.

[0091] In a specific embodiment, the first server will send the seed file to the second client only when the second client also has the permission to obtain the seed file. For the content about obtaining the seed file permission, please refer to the above, which will not be elaborated here. The level of the first client's permission to obtain the seed file and the level of the second client's permission to obtain the seed file can be the same or different; the type of the first client's permission to obtain the seed file and the type of the second client's permission to obtain the seed file can be the same or different; the access control policy for the first client's permission to obtain the seed file and the access control policy for the second client's permission to obtain the seed file can be the same or different.

[0092] In a specific embodiment, the seed file is sent to the second client only when the first server authenticates the second client. The authentication method of the first server to the second client is basically similar to the authentication method of the first server to the first client, and will not be elaborated here.

[0093] Step S107: The first client sends a first multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server receives the first multi-source download provider acquisition request sent by the first client based on the seed file.

[0094] In a specific embodiment, since the URL of the second server is recorded in the seed file, the first client can locate the second server according to the URL of the second server, thereby achieving communication between the first client and the second server.

[0095] In a specific embodiment, the first multi-source download provider acquisition request is used to acquire a list of clients for downloading the object.

[0096] In a specific embodiment, when the second server receives the first multi-source download provider acquisition request, the first client is added to the multi-source download provider list.

[0097] Step S108: The second client sends a second multi-source download provider acquisition request to the second server based on the seed file. Correspondingly, the second server receives the second multi-source download provider acquisition request sent by the second client based on the seed file.

[0098] In a specific embodiment, since the URL of the second server is recorded in the seed file, the second client can locate the second server according to the URL of the second server, thereby realizing communication between the second client and the second server.

[0099] In a specific embodiment, the second multi-source download provider acquisition request is used to obtain a list of clients for downloading the object.

[0100] In a specific embodiment, when the second server receives the second multi-source download provider acquisition request, the second client is added to the multi-source download provider list.

[0101] Step S109: The second server sends the first multi-source download provider list to the first client. Correspondingly, the first client receives the first multi-source download provider list sent by the second server.

[0102] In a specific embodiment, the second server sends the multi-source download provider list to the first client only when the first client is authenticated. The second server authenticates the first client to ensure that only trusted clients can access, use or interact with the second server, prevent unauthorized clients from accessing the second server, and protect the security and confidentiality of the multi-source download provider list.

[0103] In a specific embodiment, in order to ensure security, the second server will send the multi-source download provider list to the first client only if the first client has the permission to obtain the multi-source download provider list. The type of permission to obtain the multi-source download provider list can be public read, private, etc. Of course, in addition to the permission types for obtaining the multi-source download provider list mentioned above, the second server can also manage the permission to obtain the multi-source download provider list in a fine-grained manner through access control policies.

[0104] In a specific embodiment, the second server can send a multi-source download provider list including all multi-source download providers or a multi-source download provider list of some multi-source download providers as the first multi-source download provider list to the first client, for example, some multi-source download providers are multi-source download providers with relatively high upload rate and / or download speed, and some multi-source download providers are other multi-source download providers in the multi-source download provider list except the first client. In a more specific embodiment, the multi-source download provider list includes the second client.

[0105] In a specific embodiment, when the security requirement is relatively high, the second server needs to send a first token to the first client in addition to sending a list of multi-source download providers to the first client. The first token is used to authorize the first client to download the target object from the second client. The first token can be a long-term token or a short-term token. When the first token expires, the first client cannot download the object from the second client.

[0106] Step S110: The second server sends the second multi-source download provider list to the second client. Correspondingly, the second client receives the second multi-source download provider list sent by the second server.

[0107] In a specific embodiment, the second server sends the multi-source download provider list to the second client only when the second client is authenticated. The second server authenticates the second client to ensure that only trusted clients can access, use or interact with the second server, prevent unauthorized clients from accessing the second server, and protect the security and confidentiality of the multi-source download provider list.

[0108] In a specific embodiment, in order to ensure security, the second server will send the multi-source download provider list to the second client only when the second client has the permission to obtain the multi-source download provider list. The specific introduction to the multi-source download provider list is as mentioned above and will not be repeated here. The level of the first client's permission to obtain seed files and the level of the second client's permission to obtain seed files can be the same or different; the type of the first client's permission to obtain the multi-source download provider list and the type of the second client's permission to obtain the multi-source download provider list can be the same or different; the access control policy of the first client's permission to obtain the multi-source download provider list and the access control policy of the second client's permission to obtain the multi-source download provider list can be the same or different.

[0109] In a specific embodiment, the second server can send a multi-source download provider list including all multi-source download providers or a multi-source download provider list of some multi-source download providers to the second client. For example, some multi-source download providers are multi-source download providers with relatively high upload rate and / or download speed, and some multi-source download providers are other multi-source download providers in the multi-source download provider list except the second client. Because the first client downloads the multi-source download provider list first and the second client downloads the multi-source download provider list later, there may be other clients joining or leaving the multi-source download provider list between the two. Therefore, the multi-source download provider list downloaded by the first client and the multi-source download provider list downloaded by the second client may be different. In a more specific embodiment, the multi-source download provider list includes the first client.

[0110] In a specific embodiment, when the security requirement is relatively high, in addition to sending the multi-source download provider list to the second client, the second server also needs to send the second token to the second client. The second token is used to authorize the second client to download the object from the first client. The second token can be a long-term token or a short-term token. When the second token expires, the second client cannot download the target object from the first client. It can be understood that the first token and the second token can both be long-term tokens, or short-term tokens, or the first token can be a long-term token and the second token can be a short-term token, or the first token can be a short-term token and the second token can be a long-term token.

[0111] Step S111: the first server sends the first part of the object to the first client. Correspondingly, the first client receives the first part of the object sent by the first server.

[0112] In a specific embodiment, in order to ensure security, the first server will send the first part of the object to the first client only when the first client has the permission to obtain the object. The level of the permission to obtain the object can be any one of the permissions at the object bucket level, the object level, etc. The type of the permission to obtain the object can be public read, private, etc. Of course, in addition to the level of the object permission and the type of permission mentioned above, the first server can also manage the permission to obtain the object in a fine-grained manner through the access control policy.

[0113] In a specific embodiment, since the first server has divided the target object into multiple blocks, the first server can select some blocks from the multiple blocks and send them to the first client. For example, assuming that the first server divides the object into 4 blocks, the first server can select the first block and the third block from the 4 blocks and send them to the first client. In a more specific embodiment, when selecting which blocks to distribute to the first client, it can be implemented according to the piece selection algorithm, so as to maximize the download speed, balance upload and download, and ensure the integrity of the file. Alternatively, it can be selected according to the random selection algorithm to distribute which blocks to the first client, that is, regardless of the speed difference between the clients, some blocks are randomly selected from the multiple blocks and sent to the client; it can be selected according to the scarcity priority algorithm to distribute which blocks to the first client, that is, some scarce blocks are preferentially selected to be sent to the client, for example, there are already 5 client blocks with block A, and only 1 client block has block B, then, block B is preferentially selected to be sent. Alternatively, it can be selected according to the high-speed priority algorithm to distribute which blocks to the first client, that is, blocks provided by clients with faster download speeds are preferentially selected, and so on. Of course, the above algorithms can also be combined to select blocks from multiple blocks, so that a more scientific decision can be made. For example, a decision can be made by combining the above algorithms in a weighted manner.

[0114] Step S112: the first server sends the second part of the target object to the second client. Correspondingly, the second client receives the second part of the target object sent by the first server.

[0115] In a specific embodiment, in order to ensure security, the first server will send the second part of the object to the second client only when the second client has the permission to obtain the object. For the content about obtaining the object permission, please refer to the above, which will not be elaborated here. The level of the first client's permission to obtain the object and the level of the second client's permission to obtain the object can be the same or different; the type of the first client's permission to obtain the object and the type of the second client's permission to obtain the object can be the same or different; the access control policy of the first client's permission to obtain the object and the access control policy of the second client's permission to obtain the object can be the same or different.

[0116] In a specific embodiment, the first server may select some blocks from a plurality of blocks and send them to the second client. For example, assuming that the first server divides the target object into 4 blocks, the first server may select the second block and the fourth block from the 4 blocks and send them to the first client. It is understandable that it is assumed here that completely different blocks are sent to the first client and the second client respectively, but in actual applications, the first block and the third block may be sent to the first client, and the first block and the fourth block may be sent to the second client, or the first block and the third block may be sent to the first client, and the first block and the third block may be sent to the second client. In other words, the first part in the object and the second part in the target object may be completely identical, partially identical, or completely different, and no specific limitation is made here.

[0117] In a specific embodiment, the permission to obtain an object and the permission to obtain a seed file may be different. For example, since the confidentiality requirement of the object data is higher than that of the seed file data, the permission to obtain an object may be higher than the permission to obtain a seed file.

[0118] Step S113: The second client sends the second part of the target object to the first client. Correspondingly, the first client receives the second part of the target object sent by the second client.

[0119] In a specific embodiment, if a connection has been established between the first client and the second client before, and the connection has not been cancelled, then the second client can directly send the second part of the object to the first client. If the first client and the second client have not established a connection, or the connection between the first client and the second client has been cancelled, then the first client and the second client can establish or re-establish a connection. Since the second client has obtained a list of multi-source download providers, and the list of multi-source download providers includes the IP address and port number of the first client, etc. Therefore, the second client can locate the first client in the network through the IP address and port number of the first client, and establish a network connection with the first client. After the connection between the first client and the second client is established, the first client can send the second part of the object to the first client.

[0120] In a specific embodiment, before the second client sends the second part of the object to the first client, token verification is required. Specifically, the first client first sends a first object acquisition request to the second client, wherein the first object acquisition request carries the first token. The second client receives the first object acquisition request, extracts the first token from the first object acquisition request, and sends the first token to the second server. After receiving the first token, the second server verifies the first token, and if the verification passes, sends a first verification pass message to the second client; if the verification fails, sends a first verification failure message to the second client. After receiving the first verification pass message, the second client sends the second part of the object to the first client; after receiving the first verification failure message, the second client refuses to send the second part of the object to the first client.

[0121] Step S114: the first client sends the first part of the target object to the second client. Correspondingly, the second client receives the first part of the target object sent by the first client.

[0122] In a specific embodiment, if a connection has been established between the first client and the second client before, and the connection has not been cancelled, then the first client can directly send the first part of the target object to the second client. If the first client and the second client have not established a connection, or the connection between the first client and the second client has been cancelled, then the first client and the second client can establish or re-establish a connection. Because the first client has obtained a list of multi-source download providers, and the list of multi-source download providers includes the IP address and port number of the second client, etc. Therefore, the first client can locate the second client in the network through the IP address and port number of the second client, and establish a network connection with the second client. After the connection between the first client and the second client is established, the second client can send the first part of the target object to the first client. In addition, after the second client completes sending the first part of the target object to the first client, the connection between the first client and the second client can be cancelled to release network resources.

[0123] In a specific embodiment, before the first client sends the first part of the object to the second client, token verification is required. Specifically, the second client first sends a second object acquisition request to the first client, wherein the second object acquisition request carries the second token. The first client receives the second object acquisition request, extracts the second token from the second object acquisition request, and sends the second token to the second server. After receiving the second token, the second server verifies the second token, and if the verification passes, sends a second verification pass message to the first client; if the verification fails, sends a second verification failure message to the first client. After receiving the second verification pass message, the first client sends the first part of the target object to the second client; after receiving the second verification failure message, the first client refuses to send the first part of the target object to the second client.

[0124] In the above embodiment, it is assumed that the first client sends the first block and the third block in the object to the second client, and the second client sends the second block and the fourth block in the object to the first client. However, in actual applications, the object storage system often includes more clients, for example, the third client, so there may be more complex situations such as the first client sending the first part to the second client, the second client sending the first part to the third client, etc. There may even be a situation where the first client sends the first part to the second client, and the third client sends the second part to the second client. In addition, the multiple blocks downloaded by the first client from the first server may not all be sent to the second client, but only a part of them may be selected to be sent to the second client. For example, the first client downloads the first block and the third block from the first server, but the first client only sends the third block to the second client. In addition, in addition to sending the blocks downloaded from the first server to the second client, the first client can also send the blocks received from other clients to the second client. For example, the first client downloads the first block from the first server and the third block from the third client, and then the first client sends the first block and the third block to the second client. The client can also obtain blocks in the target object only from other clients without obtaining blocks from the first server. For example, assuming that the first server divides the target object into 4 blocks, the first server can select the first block and the third block from the 4 blocks and send them to the first client, select the second block and the fourth block from the 4 blocks and send them to the second client, and the third client obtains the first block and the third block from the first client, and obtains the second block and the fourth block from the second client.

[0125] In the above embodiments, the execution order between the steps should be determined according to the logical relationship between the steps, and should not be understood as only being determined according to the logical relationship between the steps. Figure 3For example, steps S105 and S106 may occur before step S102, step S103 or step S104, steps S105 and S106 may occur after step S107, steps S105, S106 and S108 may occur after step S109, steps S105, S106, S108 and S110 may occur after step S111, steps S105, S106, S108, S110 and S112 may occur after step S113, and so on.

[0126] Compared with the original object storage system, the relationship between the client and the first server in the original object storage system is relatively simple. The client and the first server are connected by a single line. Each client only has an upload and download relationship with the first server. Therefore, the first server only needs to simply verify whether the client has the download permission. However, in the object storage system of the present application, the relationship between the first server and the client is very complicated. Not only does the first server and the client have a data upload and download relationship, but a client theoretically has an upload and download relationship with any other client in the object storage system. In such a complex system, how to ensure the security of the object, the first server and the client is very important. Therefore, before the first server of the present application sends the seed file to the first client and the second client, before the second server sends the multi-source download provider list to the first client and the second client, and before the first server sends the object data to the first client and the second client, the first server must first authenticate the identity of the first client and the second client, confirm the authority, etc. In addition, before the first client and the second client download the data of the target object, the token must also be authenticated to ensure the security of the object download.

[0127] Because the metadata of the target object and the user data of the object may be stored in different places, assuming that the user data of the target object is stored in Beijing, the metadata of the target object is stored in Guangzhou, and when the client in Shanghai needs to download the target object, it needs to first obtain the metadata of the target object from the server in Guangzhou, and then obtain the user data of the target object from the server in Beijing, which will cause a relatively large delay. For example, assuming that the delay for the server in Shanghai to obtain the metadata of the target object from the server in Guangzhou is 10 seconds, and the delay for obtaining the user data of the target object from the server in Beijing is 1 second, then the total delay is 11 seconds. However, if the seed file downloaded by the client contains the metadata of the target object, then when the client in Shanghai needs to download the target object, it only needs to obtain the user data of the target object from the server in Beijing. Taking the above example as an example, the total delay at this time is 1 second, so it can greatly reduce the delay and improve the user experience. In the above example, it is assumed that the client, the server storing metadata, and the server storing user data are located in different areas. In actual applications, the client, the server storing metadata, and the server storing user data can also be partially located in the same area, or all located in the same area.

[0128] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an object storage system provided by this application. Figure 5 As shown, the object storage system of the present application includes: a client 210 , a first server 220 and a second server 230 .

[0129] The client 210 includes a seed acquisition unit 211 , a list acquisition unit 212 , an object acquisition unit 213 , a receiving unit 214 , and a sending unit 215 .

[0130] The seed acquisition unit 211 is used to instruct the sending unit 215 to send a seed acquisition request to the first server 220, and receive the seed file sent by the first server 220 through the receiving unit 214. After obtaining the seed file, the seed acquisition unit 211 will notify the list acquisition unit 212 that it needs to obtain the list of multi-source download providers. The seed acquisition unit 211 can be used to provide the following functions: seed file search function, users can search and browse various seed files by keywords; download files from multiple sources at the same time, thereby significantly improving download speed and efficiency; manage multiple download tasks, including setting download queues, priorities and the number of tasks to be downloaded at the same time, so as to better control the download order and speed; manage downloaded seed files, including operations such as adding, deleting, pausing, resuming and re-downloading; support encrypted transmission and anonymous download functions to increase user privacy protection and security, etc.

[0131] The list acquisition unit 212 is used to instruct the sending unit 215 to send a multi-source download provider acquisition request to the second server 230, and receive the multi-source download provider list sent by the second server 230 through the receiving unit 214. After acquiring the multi-source download provider acquisition request, the list acquisition unit 212 will notify the object acquisition unit 214 that the target object needs to be acquired.

[0132] The object acquisition unit 213 is used to instruct the receiving unit 214 to receive a part of the target object sent by the first server 220, to receive a part of the target object from other clients; or, to receive all the target objects from one or more other clients. When the object acquisition unit 213 receives the target object from one or more other clients, it can receive a part of the target object sent by client A, a part of the target object sent by client B, and so on. The object acquisition unit 213 can have the following functions: upload local files to the object storage service of the first server 220; combine the parts of the target object downloaded from different sources into an object; manage the uploaded target object, including operations such as creation, deletion, copying, moving, and renaming; set the access rights of the file, including public read and write, private read and write, etc., to ensure the security and privacy of the file; support multiple upload methods, including single file upload, batch upload, and breakpoint resume, etc., to provide a more flexible and efficient upload experience; provide data encryption function, encrypt the uploaded object to increase the security of the data; support data backup and recovery functions, users can back up important data to the object storage service and restore it when needed.

[0133] The seed acquisition unit 211 may instruct the sending unit 215 to execute Figure 4 The step of the first client in step S101 of the object downloading method shown in FIG. 100 , or the step of instructing the sending unit 215 to execute Figure 4 The seed acquisition unit 211 may instruct the receiving unit 214 to perform Figure 4 The step performed by the first client in step S104 of the object downloading method shown in FIG. 1 , or the step instructing the receiving unit 214 to perform Figure 4 The step S106 of the object downloading method shown is performed by the second client.

[0134] The list acquisition unit 212 may instruct the sending unit 215 to execute Figure 4 The step of the first client executing in step S107 of the object downloading method shown in FIG. 1 , or instructing the sending unit 215 to execute Figure 4 The step performed by the second client in step S108 of the object downloading method shown in FIG. 21. The list acquisition unit 212 may instruct the receiving unit 214 to perform Figure 4The step of the first client executing in step S109 of the object downloading method shown in FIG. 1 , or instructing the receiving unit 214 to execute Figure 4 The step S110 of the object downloading method shown is performed by the second client.

[0135] The object acquisition unit 213 may instruct the receiving unit 214 to execute Figure 4 The steps of the object download method shown in step S111 and S113 executed by the first client, or the receiving unit 214 is instructed to execute Figure 4 The steps S112 and S114 of the object downloading method shown are steps performed by the second client.

[0136] It should be understood that Figure 5 The client in is an exemplary division method, and the client 210 may also include more or fewer units. For example, the client 210 may also include a display unit for displaying seed files, lists of multi-source download providers, or objects. For another example, the seed acquisition unit 211, the list acquisition unit 212, and the object acquisition unit 213 may be combined into one acquisition unit. For another example, the client 210 may not have a display unit. It should be understood that the above examples are for illustration, and the specific units may be determined according to the actual application scenario, and this application does not make specific limitations.

[0137] In order to make this application better understood, Figure 5 Only one client's internal structure is shown. In fact, the number of clients can be one or more. The internal structures of other clients are similar to Figure 5 The internal structure of the client 210 shown is the same and will not be described in detail here.

[0138] The first server 220 includes a receiving unit 221 , a sending unit 222 , a seed generating unit 223 , and an object storing unit 224 .

[0139] The seed generating unit 223 is used to receive the seed obtaining request sent by the client through the receiving unit 221, generate a seed file for the object, and instruct the sending unit 222 to send the seed file to the second server for management.

[0140] The object storage unit 224 is used to instruct the sending unit 222 to send the seed file to the second server and send the object to the client. The object storage unit may include the following functions: divide the object into multiple parts; store large-scale data objects and support the storage and management of massive data; ensure the security and reliability of data through data redundancy and backup mechanisms; provide high-availability storage services with distributed architecture and data replication mechanisms; provide multiple access interfaces and protocols, such as RESTful API, S3 API, etc., so that users can easily access and manage stored data; provide data security protection and encryption functions. It can encrypt and store data to protect the confidentiality of data. At the same time, it also provides access control and permission management to ensure that only authorized users can access and modify data; provide data management and retrieval functions, such as data classification, indexing, search, etc.; support cross-regional replication and backup of data; verify the integrity of downloaded data to ensure that the downloaded object is completely consistent with the original object.

[0141] The seed generation unit 223 can be executed by the receiving unit 221 Figure 4 The steps performed by the first server in steps S101 and S105 of the object downloading method shown in FIG. Figure 4 Step S102 of the object downloading method shown in FIG. 102 instructs the sending unit 222 to execute Figure 4 The steps S103, S104 and S106 of the object downloading method shown are steps executed by the first server.

[0142] The object storage unit 224 instructs the sending unit 222 to execute Figure 4 The steps S111 and S112 of the object downloading method shown are steps executed by the first server.

[0143] It should be understood that Figure 5 The first server 220 in the figure is an exemplary division method. The first server 220 may also include more or fewer units. For example, the first server 220 may also include a display unit for displaying a seed file or an object. For another example, the seed generation unit 223 and the object storage unit 224 may be combined into one acquisition unit. For another example, the first server 220 may not have a display unit. It should be understood that the above examples are for illustration purposes only. The specific units may be determined according to the actual application scenario, and this application does not make any specific limitations.

[0144] The seed generation unit 223 and the object storage unit 224 can be implemented by software or hardware. For example, the implementation of the object storage unit 224 is described below by taking the object storage unit 224 as an example. Similarly, the implementation of the seed generation unit 223 can refer to the implementation of the object storage unit 224.

[0145] As an example of a software functional unit, the object storage unit 224 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the object storage unit 224 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0146] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0147] As an example of a hardware functional unit, the object storage unit 224 may include at least one computing device, such as a server, etc. Alternatively, the object storage unit 224 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0148] The multiple computing devices included in the object storage unit 224 can be distributed in the same region or in different regions. The multiple computing devices included in the object storage unit 224 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the object storage unit 224 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0149] It should be noted that, in other embodiments, the object storage unit 224 may be used to execute Figure 4 The seed generation unit 223 can be used to execute any step of the first server in the object download method shown in the figure. The steps that the seed generation unit 223 and the object storage unit 224 are responsible for implementing can be specified as needed. The seed generation unit 223 and the object storage unit 224 respectively implement different steps in the object download method to realize all the functions of the first server.

[0150] The second server 230 includes a receiving unit 231 , a sending unit 232 , and a list management unit 233 .

[0151] The list management unit 233 receives the seed file sent by the first server 220 through the receiving unit 231, receives the multi-source download provider acquisition request sent by the client, and instructs the sending unit 232 to send the multi-source download provider list. The functions of the list management unit 233 may include: saving a large number of seed files; coordinating data transmission between various clients; recording the status and progress of each download task, and providing this information to other clients; obtaining the IP address and port number of other clients to establish a point-to-point data transmission connection; coordinating data transmission between various clients, and scheduling according to the upload and download speed of each client to optimize the performance and efficiency of the entire download system.

[0152] The list management unit 233 can be executed by the receiving unit 231 Figure 4 The steps performed by the second server in steps S103 and S107 of the object downloading method shown in the figure instruct the sending unit 232 to perform Figure 4 The steps S109 and S110 of the object downloading method shown are steps executed by the second server.

[0153] It should be understood that Figure 5The second server 230 in the figure is an exemplary division method. The second server 230 may also include more or fewer units. For example, the second server 230 may also include a display unit for displaying a list of multi-source download providers. For another example, the receiving unit 231 and the sending unit 232 may be combined into one acquisition unit. For another example, the second server 230 may not have a display unit. It should be understood that the above examples are for illustration purposes only. The specific units may be determined according to the actual application scenario, and this application does not make any specific limitations.

[0154] The list management unit 233 may be implemented by software or hardware. For example, the object storage unit 224 may be implemented by software or hardware.

[0155] As an example of a software functional unit, the list management unit 233 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the list management unit 233 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0156] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0157] As an example of a hardware functional unit, the list management unit 233 may include at least one computing device, such as a server, etc. Alternatively, the list management unit 233 may also be a device implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0158] The multiple computing devices included in the list management unit 233 can be distributed in the same region or in different regions. The multiple computing devices included in the list management unit 233 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the list management unit 233 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0159] It should be noted that, in other embodiments, the list management unit 233 may be used to execute Figure 4 Any steps of the second server in the object downloading method shown, the steps that the list management unit 233 is responsible for implementing can be specified as needed.

[0160] The present application also provides a computing device 400. Figure 6 As shown, computing device 400 includes: bus 402, processor 404, memory 406 and communication interface 408. Processor 404, memory 406 and communication interface 408 communicate through bus 402. Computing device 400 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 400.

[0161] The bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6The bus 402 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 402 may include a path for transmitting information between various components of the computing device 400 (eg, the memory 406, the processor 404, and the communication interface 408).

[0162] The processor 404 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0163] The memory 406 may include a volatile memory, such as a random access memory (RAM). The processor 404 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0164] When the computing device is a client, the memory 406 stores executable program codes, and the processor 404 executes the executable program codes to respectively implement the functions of the aforementioned seed acquisition unit, the list acquisition unit, and the object acquisition unit, thereby implementing the steps executed by the client in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the client in the object download method. Alternatively, the memory 406 stores executable codes, and the processor 404 executes the executable codes to respectively implement the functions of the aforementioned client, thereby implementing the instructions for the steps executed by the client in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the client in the object download method.

[0165] When the computing device is the first server, the memory 406 stores an executable program code, and the processor 404 executes the executable program code to respectively implement the functions of the aforementioned seed generation unit and the object storage unit, thereby implementing the steps executed by the first server in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the first server in the object download method. Alternatively, the memory 406 stores executable code, and the processor 404 executes the executable code to respectively implement the functions of the aforementioned first server, thereby implementing the instructions for the steps executed by the first server in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the first server in the object download method.

[0166] When the computing device is a second server, the memory 406 stores executable program codes, and the processor 404 executes the executable program codes to respectively implement the functions of the aforementioned list management unit, thereby implementing the steps executed by the second server in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the second server in the object download method. Alternatively, the memory 406 stores executable codes, and the processor 404 executes the executable codes to respectively implement the functions of the aforementioned second server, thereby implementing the steps executed by the second server in the object download method. That is, the memory 406 stores instructions for executing the steps executed by the second server in the object download method.

[0167] The communication interface 408 uses a receiving unit such as, but not limited to, a network interface card, a transceiver, and a sending unit to implement communication between the computing device 400 and other devices or a communication network.

[0168] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0169] like Figure 7 As shown, the computing device cluster includes at least one computing device 400. The memory 406 in one or more computing devices 400 in the computing device cluster may store the same Figure 4 Instructions for the steps executed from any angle by the first server and the second server in the object downloading method.

[0170] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the steps executed by any angle of the first server and the second server in the object downloading method. In other words, the combination of one or more computing devices 400 can jointly execute instructions for executing the steps executed by any angle of the first server and the second server in the object downloading method.

[0171] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the first server or the second server. That is, the instructions stored in the memory 406 in different computing devices 400 can implement the functions of one or more units in the seed generation unit 223 and the object storage unit 224, or implement the function of the list management unit.

[0172] In some possible implementations, taking the first server as an example, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 8 A possible implementation is shown. Figure 8 As shown, two computing devices 400A and 400B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 406 in the computing device 400A stores instructions for executing the functions of the seed generation unit 223 in the first server. At the same time, the memory 406 in the computing device 400B stores instructions for executing the functions of the object storage unit 224.

[0173] Figure 8 The connection method between the computing device clusters shown may be based on the needs of the object downloading method provided in the present application (eg, storing a large amount of data), and therefore, it is considered that the functions implemented by the object storage unit 224 are handed over to the computing device 400B for execution.

[0174] It should be understood that Figure 8 The functions of the computing device 400A shown in FIG. 4 may also be completed by multiple computing devices 400. Similarly, the functions of the computing device 400B may also be completed by multiple computing devices 400.

[0175] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 7 and Figure 8The connection mode of the computing device cluster is different in that the memory 406 in one or more computing devices 400 in the computing device cluster may store the same instructions for executing the steps executed by the first server in the object downloading method.

[0176] In some possible implementations, the memory 406 of one or more computing devices 400 in the computing device cluster may also store partial instructions for executing the first server in the object download method. In other words, the combination of one or more computing devices 400 may jointly execute instructions for executing the steps executed by the first server in the object download method.

[0177] It should be noted that the memory 406 in different computing devices 400 in the computing device cluster can store different instructions for respectively executing the functions of the first server and the second server. That is, the instructions stored in the memory 406 in different computing devices 400 can implement the functions of one or more servers in the first server and the second server.

[0178] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Figure 4 The object downloading method shown includes steps performed by one or more of the client, the first server, or the second server.

[0179] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute one or more steps performed by the client, the first server, or the second server in the object download method.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0181] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

Claims

1. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first client sends a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; The first server sends the seed file to the first client based on the first seed acquisition request; The first client sends a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server sends a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The first client receives a first part of the target object sent by the first server, and receives a second part of the target object sent by the second client.

2. The method according to claim 1, characterized in that The method further comprises: When the first client is authenticated successfully, the second server sends the first token to the first client; Before receiving the second part of the target object sent by the second client, the method further includes: The first client sends the first token to the second client; The second client sends the first token to a second server; The second server verifies the first token, and if the verification is successful, sends a first verification success message to the second client; After receiving the first verification success message, the second client sends the second part of the target object to the first client.

3. The method according to claim 2, characterized in that Before sending the second part of the target object to the first client, the method further includes: The second client sends a second seed acquisition request to the first server, wherein the second seed acquisition request is used to request a seed file of the target object; The second client runs the seed file and sends a second multi-source download provider acquisition request to the second server, wherein the second multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server authenticates the second client based on the second multi-source download provider acquisition request, and sends a second multi-source download provider list to the second client if the second client passes the authentication; The second client receives a second part of the target object sent by the first server.

4. The method according to claim 3, characterized in that The method further comprises: When the second client is authenticated successfully, the second server sends a second token to the second client; Before receiving the first part of the target object sent by the first client, the method further includes: The second client sends the second token to the first client; The first client sends the second token to the second server; The second server verifies the second token, and if the verification is successful, sends a second verification success message to the first client; After receiving the second verification success message, the first client sends the first part of the target object to the second client.

5. The method according to any one of claims 1 to 4, characterized in that: The first server is used to open a first permission for obtaining the seed file and a second permission for obtaining the target object to the first client, wherein the first permission is lower than the second permission.

6. The method according to any one of claims 1 to 5, characterized in that: The first server includes a metadata server and a user data server, and the target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server.

7. The method according to claim 6, characterized in that The seed file includes metadata of the target object.

8. The method according to any one of claims 1 to 7, characterized in that: The first server sends the seed file to the first client based on the seed acquisition request, including: The first server creates the seed file for the target object based on the seed acquisition request, and sends the seed file to the first client.

9. The method according to any one of claims 1 to 8, characterized in that: The second server sends the first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, including: The second server authenticates the first client based on the first multi-source download provider acquisition request, and sends a first multi-source download provider list to the first client if the first client passes the authentication.

10. An object downloading method, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first client sends a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; Receiving the seed file sent by the first server based on the first seed acquisition request; Sending a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the multi-source download of the target object for the first client; Receiving a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; A first part of the target object sent by the first server is received, and a second part of the target object sent by the second client is received.

11. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The first server receives a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object; receiving a second seed acquisition request sent by the second client, wherein the second seed acquisition request is used to request a seed file of the target object; Sending the seed file to the first client and the second client respectively; Sending a first portion of the target object to the first client; A second portion of the target object is sent to the second client.

12. A method for downloading an object, characterized in that: The method is applied to an object storage system, the object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, the method includes: The second server receives the seed file of the target object sent by the first server, and establishes a multi-source download provider list based on the seed file; Receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with downloads of the target object; Based on the first multi-source download provider acquisition request, a first multi-source download provider list is sent to the first client, the first multi-source download provider list includes the second client, and the first multi-source download provider list belongs to the multi-source download provider list.

13. An object storage system, characterized in that The object storage system includes a first client, a second client, a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, The first client is used to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; The first server is used for sending the seed file to the first client based on the first seed acquisition request; The first client is used to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The first client is used to receive the first part of the target object sent by the first server, and receive the second part of the target object sent by the second client.

14. The system according to claim 13, characterized in that The second server is further configured to send a first token to the first client when the first client is authenticated successfully; The first client is further used to send the first token to the second client; The second client is further used to send the first token to a second server; The second server is further used to verify the first token, and if the verification is successful, send a first verification success message to the second client; The second client is further configured to send the second part of the target object to the first client after receiving the first verification success message.

15. The system according to claim 14, characterized in that The second client is used to send a second seed acquisition request to the first server, wherein the second seed acquisition request is used to request a seed file of the target object; The second client is used to run the seed file and send a second multi-source download provider acquisition request to the second server, wherein the second multi-source download provider acquisition request is used to request a list of at least one client for downloading the target object; The second server is used to authenticate the second client based on the second multi-source download provider acquisition request, and send a second multi-source download provider list to the second client if the second client passes the authentication, wherein the second multi-source download provider list includes the first client; The second client is used to receive the second part of the target object sent by the first server.

16. The system according to claim 15, characterized in that The second server is further configured to send a second token to the second client if the second client is authenticated successfully; The second client is further used to send the second token to the first client; The first client is further used to send the second token to the second server; The second server is further used to verify the second token, and if the verification is successful, send a second verification success message to the first client; The first client is further configured to send the first part of the target object to the second client after receiving the second verification success message.

17. The system according to any one of claims 13 to 16, characterized in that: The first server is used to open a first permission for obtaining the seed file and a second permission for obtaining the target object to the first client, wherein the first permission is lower than the second permission.

18. The system according to any one of claims 13 to 17, characterized in that: The first server includes a metadata server and a user data server, and the target object includes metadata and user data, wherein the metadata is stored in an object storage device managed by the metadata server, and the user data is stored in an object storage device managed by the user data server.

19. The system according to claim 18, characterized in that The seed file includes metadata of the target object.

20. The system according to any one of claims 13 to 19, characterized in that: The first server is used to create the seed file for the target object based on the seed acquisition request, and send the seed file to the first client.

21. The system according to any one of claims 13 to 20, characterized in that: The second server is used for authenticating the first client based on the first multi-source download provider acquisition request, and sending the first multi-source download provider list to the first client if the first client is authenticated successfully.

22. A first client, wherein: The first client belongs to an object storage system, the object storage system further includes a second client, a first server and a second server, cloud computing includes a first server and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first client includes: A sending unit, configured to send a first seed acquisition request to the object storage service on the first server, wherein the first seed acquisition request is used to request a seed file of a target object; A receiving unit, configured to receive the seed file sent by the first server based on the first seed acquisition request; The sending unit is used to send a first multi-source download provider acquisition request to the second server based on the seed file, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the multi-source download of the target object for the first client; The receiving unit is used to receive a first multi-source download provider list sent by the second server based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client; The receiving unit is used to receive a first part of the target object sent by the first server, and receive a second part of the target object sent by the second client.

23. A first server, characterized in that: The first server belongs to an object storage system, the object storage system further includes a first client, a second client and a second server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the first server includes: A receiving unit, configured to receive a first seed acquisition request sent by a first client, wherein the first seed acquisition request is used to request a seed file of a target object; The receiving unit is further configured to receive a second seed acquisition request sent by a second client, wherein the second seed acquisition request is used to request a seed file of the target object; A sending unit, configured to send the seed file to the first client and the second client respectively; The sending unit is further configured to send the first part of the target object to the first client; The sending unit is further configured to send the second part of the target object to the second client.

24. A second server, characterized in that: The second server belongs to an object storage system, the object storage system further includes a first client, a second client and a first server, the first server deploys an object storage service, the first server and the second server are the same server or different servers, and the second server includes: A receiving unit, configured to receive a seed file of a target object sent by the first server, and to establish a multi-source download provider list based on the seed file; The receiving unit is further configured to receive a first multi-source download provider acquisition request sent by a first client, and add the first client to the multi-source download provider list, wherein the first multi-source download provider acquisition request is used to request a list of clients that can provide the first client with downloads of the target object; The sending unit is used to send a first multi-source download provider list to the first client based on the first multi-source download provider acquisition request, wherein the first multi-source download provider list includes the second client and the first multi-source download provider list belongs to the multi-source download provider list.

25. A computer device, characterized in that: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

26. A computing cluster, characterized in that: The computing cluster includes a plurality of computer devices, each of which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method according to claims 1-9 when executing the computer program.

27. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to claims 1 to 9.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when executed on a computer device, enable the computer device to execute the method according to any one of claims 1 to 9.

Images