File downloading method and device, computer equipment and readable storage medium

Abstract

The invention relates to a file downloading method and device, computer equipment and a readable storage medium. The method comprises the following steps: receiving a file downloading request sent by a client; the file downloading request comprises a storage position index corresponding to a to-be-downloaded file and a file identifier; according to the storage position index, a first mounting directory corresponding to the storage position index is matched from a plurality of distributed storage node mounting directories; according to the file identifier, accessing a distributed storage node corresponding to the first mounting directory through the first mounting directory, determining a to-be-downloaded file pointed by the file identifier, and sending the to-be-downloaded file to the client; the problem that in the prior art, the file downloading efficiency is low due to the fact that an intermediate link is too long during file downloading is solved, file downloading is achieved through the gateway agent layer, the intermediate link in the file downloading process can be effectively shortened, and the file downloading efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of data downloading, and in particular to a file downloading method, device, computer equipment and readable storage medium. Background Art

[0002] With the diversification of Internet channels and the continuous enrichment of terminal device connection methods, file downloading has become one of the indispensable functions of Internet services. However, facing the geometric growth trend of the number of users, it is particularly important to provide efficient and fast file download services. Currently, many solutions mainly rely on centralized cluster architecture to support high-speed file download requirements, but this approach faces a series of challenges, especially the problem of low file download efficiency caused by long intermediate links.

[0003] There is no effective solution to the problem of low file downloading efficiency in the prior art. Summary of the invention

[0004] Based on this, it is necessary to provide a file downloading method, device, computer equipment and readable storage medium to address the above technical problems.

[0005] In a first aspect, the present application provides a file downloading method, which is applied in a gateway proxy layer, and the method comprises: Receive a file download request sent by a client; the file download request includes a storage location index and a file identifier corresponding to the file to be downloaded; According to the storage location index, a first mount directory corresponding to the storage location index is matched from multiple distributed storage node mount directories; According to the file identifier, the distributed storage node corresponding to the first mount directory is accessed through the first mount directory to determine the file to be downloaded pointed to by the file identifier, and the file to be downloaded is sent to the client.

[0006] In one embodiment, before receiving the file download request sent by the client, the method further includes: Obtain a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; For each file to be uploaded in the set of files to be uploaded, obtaining a file uploading order of the file to be uploaded in the set of files to be uploaded; Performing a modulo process on the file upload order to determine the target hash slot; According to the preset hash slot routing table and the target hash slot, a target storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the target storage node; The target hash slot is saved as a storage location index corresponding to the file to be uploaded.

[0007] In one embodiment, performing a modulo process on the file upload order to determine a target hash slot includes: Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the file upload order is modulo processed to determine the target hash slot.

[0008] In one embodiment, obtaining the total number of hash slots corresponding to the plurality of distributed storage nodes includes: Determine a plurality of hash slots corresponding to each of the plurality of distributed storage nodes according to the storage space capacity corresponding to each of the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each of the distributed storage nodes is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0009] In one embodiment, the method further comprises: Generate distributed storage node mount directories corresponding to multiple distributed storage nodes; Mount each of the distributed storage nodes to the corresponding distributed storage node mount directory.

[0010] In one embodiment, the step of accessing a distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client includes: If the file to be downloaded corresponding to the file identifier exists in the distributed storage node corresponding to the first mounting directory, the file to be downloaded is read from the first mounting directory and sent to the client.

[0011] In one embodiment, the step of accessing the distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client further includes: If the distributed storage node corresponding to the first mount directory does not have the file to be downloaded corresponding to the file identifier, iteratively matching the mount directories of multiple distributed storage nodes in a preset iteration order to determine a second mount directory; the distributed storage node corresponding to the second mount directory stores the file to be downloaded; The file to be downloaded corresponding to the file identifier is read from the second mounting directory and sent to the client.

[0012] In a second aspect, the present application further provides a file downloading device, the device comprising: A receiving module, configured to receive a file download request sent by a client; the file download request includes a storage location index and a file identifier corresponding to the file to be downloaded; A matching module, configured to match a first mount directory corresponding to the storage location index from a plurality of distributed storage node mount directories according to the storage location index; The download module is used to access the distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client.

[0013] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the embodiments of the first aspect are implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the embodiments of the first aspect above are implemented.

[0015] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect above.

[0016] The above-mentioned file downloading method, device, computer equipment and readable storage medium; receive a file download request sent by a client, wherein the file download request includes a storage location index and a file identifier corresponding to the file to be downloaded; and then, based on the storage location index, quickly match a first mount directory corresponding to the storage location index from multiple distributed storage node mount directories, effectively improving the search efficiency of the storage location of the file to be downloaded; further, based on the file identifier, the gateway proxy layer can directly access the distributed storage node corresponding to the first mount directory through the first mount directory, and then determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client, avoiding the problem of low file download efficiency caused by the long intermediate link when downloading files in the prior art. File download is realized through the gateway proxy layer, which can effectively shorten the intermediate link of the file download process and effectively improve the efficiency of file download. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 An application environment diagram of a file downloading method in an embodiment; Figure 2 A schematic diagram of a process flow of a file downloading method in one embodiment; Figure 3 A schematic diagram of a flow chart of a storage location index determination step in one embodiment; Figure 4 A schematic diagram of the mounting relationship between a distributed storage node mounting directory and a distributed storage node in one embodiment; Figure 5 A schematic diagram of a flow chart of a file downloading method in a specific embodiment; Figure 6 A structural block diagram of a file downloading device in an embodiment; Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] The file download method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and Internet of Things devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0021] In an exemplary embodiment, Figure 2 As shown, Figure 21 is a flowchart of a file download method in an embodiment; this embodiment is illustrated by applying the method to a gateway proxy layer, wherein the gateway proxy layer may include, but is not limited to, a proxy server; in this embodiment, the method includes the following steps: Step S201: receiving a file download request sent by a client.

[0022] The file download request includes a storage location index and a file identifier corresponding to the file to be downloaded.

[0023] The storage location index corresponding to the file to be downloaded is used to represent the physical storage location corresponding to the file to be uploaded; the physical storage location is a distributed storage node. The distributed storage node can be a distributed server. The file identifier is a unique identifier corresponding to the file to be downloaded.

[0024] In an exemplary embodiment, a file download request sent by a client is received, wherein the storage location index corresponding to the file to be downloaded included in the file download request is "1000", and the file identifier corresponding to the file to be downloaded is "ABCDFS8989989699JHGFN".

[0025] Step S202: According to the storage location index, a first mount directory corresponding to the storage location index is matched from multiple distributed storage node mount directories.

[0026] Among them, the distributed storage node mount directory corresponds to the distributed storage node one by one; the distributed storage node mount directory is used to directly access the distributed storage node.

[0027] It can be understood that the distributed storage node mount directory on the gateway proxy layer maps the storage resources of the distributed storage node to a specified path in the local gateway proxy layer; based on this, the gateway proxy layer can directly access the storage resources of the distributed storage node just like accessing local files.

[0028] The distributed storage node may be a distributed server; the distributed server includes a solid-state hard disk for providing storage space. It should be noted that the number of distributed storage nodes needs to be set according to actual needs and is not specifically limited here. Each distributed storage node has a unique corresponding distributed storage node mount directory.

[0029] The first mount directory refers to a distributed storage node mount directory corresponding to the storage location index from among the multiple distributed storage node mount directories.

[0030] Step S203: According to the file identifier, the distributed storage node corresponding to the first mount directory is accessed through the first mount directory to determine the file to be downloaded pointed to by the file identifier, and the file to be downloaded is sent to the client.

[0031] Exemplarily, a file download request sent by a client is received through a gateway proxy layer, and according to the storage location index included in the file download request, a first mount directory corresponding to the storage location index can be quickly matched from multiple distributed storage node mount directories; further, since the file identifier corresponding to each file to be downloaded is unique, according to the file identifier in the file download request, the distributed storage node corresponding to the first mount directory can be directly accessed through the first mount directory to determine the file to be downloaded pointed to by the file identifier, and further, the file to be downloaded can be sent directly to the client through the gateway proxy layer, effectively shortening the intermediate link of file download.

[0032] In this embodiment, according to the storage location index, the first mount directory corresponding to the storage location index can be quickly matched from multiple distributed storage node mount directories, effectively improving the search efficiency of the storage location of the file to be downloaded; further, according to the file identifier, the gateway proxy layer can directly access the distributed storage node corresponding to the first mount directory through the first mount directory, and then determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client, avoiding the problem of low file download efficiency caused by the long intermediate link when downloading files in the prior art; realizing file download through the gateway proxy layer can effectively shorten the intermediate link of the file download process and effectively improve the efficiency of file download.

[0033] In one embodiment, Figure 3 As shown, Figure 3 The flowchart of the storage location index determination step in one embodiment is as follows; before receiving the file download request sent by the client, the following steps are also included: Step S301, obtaining a set of files to be uploaded.

[0034] The file set to be uploaded includes multiple files to be uploaded. It should be noted that the file set to be uploaded may be a large batch high-concurrency file set; each file to be uploaded in the file set to be uploaded comes from the client.

[0035] Step S302: for each file to be uploaded in the file set to be uploaded, obtain the file upload order of the file to be uploaded in the file set to be uploaded.

[0036] Among them, the file upload order refers to the upload order number of the file to be uploaded in the file set to be uploaded; for example, assuming that the file set to be uploaded includes 1000 files to be uploaded, the file upload order includes 1-1000, among which the file upload order corresponding to the first file to be uploaded is 1, the file upload order corresponding to the second file to be uploaded is 2, and so on. Each file to be uploaded has a unique corresponding file upload order.

[0037] Step S303, performing a modulo process on the file upload order to determine the target hash slot.

[0038] The target hash slot is used to determine the storage location of the file to be uploaded.

[0039] Optionally, the method of performing modulo processing on the file upload order may be, but is not limited to, mapping changes to the file upload order to determine the corresponding target hash slot.

[0040] Step S304, according to the preset hash slot routing table and the target hash slot, a target storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the target storage node.

[0041] The first storage node refers to a distributed storage node corresponding to the target hash slot among multiple distributed storage nodes.

[0042] The preset hash slot routing table includes a mapping relationship between hash slots and distributed storage nodes, and is used to quickly locate the first storage node corresponding to the target hash slot. It should be noted that the preset hash slot routing table needs to be set according to the number of multiple distributed storage nodes and the storage performance of each distributed storage node, which is not specifically limited here.

[0043] It should be noted that based on the preset hash slot routing table, when the distributed storage nodes need to be horizontally expanded, it is only necessary to add the mapping relationship between the expanded distributed storage nodes and their corresponding hash slots on the basis of the original preset hash slot routing table. There is no need to adjust the original mapping relationship. Based on this, the entire system can be easier to maintain and expand.

[0044] In an exemplary embodiment, the file distribution system includes a client, a gateway proxy layer, and three distributed storage nodes; the preset hash slot routing table is shown in Table 1, and there is a mapping relationship between distributed storage node 1 and hash slots 1-5; there is a mapping relationship between distributed storage node 2 and hash slots 6-11; there is a mapping relationship between distributed storage node 1 and hash slots 12-18. When expanding the distributed storage nodes, for example, expanding distributed storage node 4 in the system, at this time, it is only necessary to add a mapping relationship between distributed storage node 4 and hash slots 19-25 based on the original preset hash slot routing table.

[0045] Table 1 Preset hash slot routing table

[0046] It should be noted that based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes can be achieved, and at the same time, the resources of each distributed storage node can be maximized and rationalized. Among them, the decoupling of the file to be uploaded and the distributed storage nodes means that even if a distributed storage node fails or needs maintenance, only the preset hash slot routing table needs to be updated, and there is no need to adjust the file upload logic; based on the above decoupling relationship, the architecture of the entire system can be made more flexible and easy to expand horizontally.

[0047] Step S305: Save the target hash slot as a storage location index corresponding to the file to be uploaded.

[0048] Exemplarily, for any file to be uploaded in the file set to be uploaded, the file upload order of the file to be uploaded in the file set to be uploaded is obtained; further, the file upload order is modulo processed to obtain the target hash slot; according to the preset hash slot routing table, the first storage node corresponding to the file to be uploaded can be accurately determined; for example, taking Table 1 as an example, assuming that the target hash slot corresponding to the file to be uploaded is 13, according to the preset hash slot routing table in Table 1, it can be determined that the first storage node corresponding to the target hash slot 13 is the distributed storage node 3, and then the file to be uploaded can be uploaded to the distributed storage node 3, and at the same time, the target hash slot 13 is saved as the storage location index corresponding to the file to be uploaded, so that the storage location corresponding to the file to be downloaded can be quickly located when the file is downloaded.

[0049] In the present embodiment, for each file to be uploaded in the set of files to be uploaded, the target hash slot is determined by taking the modulus of the file upload order of the files to be uploaded, and according to the preset hash slot routing table and the target hash slot, multiple files to be uploaded can be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner. Based on this, the loads of multiple distributed storage nodes are balanced, the utilization rate of the resources of each distributed storage node is improved, and the load imbalance in the prior art when uploading large batches of files, that is, the overload of a single or a few storage nodes, is avoided, and the balance of file upload is effectively improved; based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes is achieved, which lays a foundation for improving the flexibility of the horizontal expansion of the distributed storage nodes. At the same time, the target hash slot corresponding to the file to be uploaded is saved as a storage location index, which lays a foundation for quickly locating the storage location corresponding to the file to be downloaded when downloading the file.

[0050] In one embodiment, performing a modulo process on the file upload order to determine the target hash slot includes the following steps: Step 1: Get the total number of hash slots corresponding to multiple distributed storage nodes.

[0051] Among them, the total number of hash slots corresponding to the distributed storage nodes is the sum of the number of hash slots corresponding to all distributed storage nodes; illustratively, taking Table 1 as an example, assuming that the file distribution system includes 3 distributed storage nodes, the amount of hash slot data corresponding to distributed storage node 1 is 5, the amount of hash slot data corresponding to distributed storage node 2 is 6, and the amount of hash slot data corresponding to distributed storage node 3 is 7. Further, by summing up the amount of hash slot data of each distributed storage node, it can be determined that the total number of hash slots is 18.

[0052] Step 2: According to the total number of hash slots, perform modulo processing on the file upload order to determine the target hash slot.

[0053] Exemplarily, according to the total number of hash slots corresponding to multiple distributed storage nodes, the order of file uploads is modulo processed, and the specific process of determining the target hash slot is as follows:

[0054] Wherein, N represents the file upload order of the to-be-uploaded files in the to-be-uploaded file set; X represents the total number of hash slots corresponding to multiple distributed storage nodes; and Y represents the target hash slot.

[0055] Preferably, step 1, obtaining the total number of hash slots corresponding to the multiple distributed storage nodes, includes the following steps: Step 1.1, according to the storage space capacity corresponding to each distributed storage node in the multiple distributed storage nodes, determine the multiple hash slots corresponding to each distributed storage node.

[0056] The storage space capacity refers to the effective storage space size corresponding to the distributed storage node; for example, the storage space size of the solid-state disk in the distributed server.

[0057] Step 1.2, summing the number of multiple hash slots corresponding to each distributed storage node to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0058] In an exemplary embodiment, a method for determining multiple hash slots corresponding to each distributed storage node according to the storage space capacity corresponding to each distributed storage node among multiple distributed storage nodes may be: for each distributed storage node among multiple distributed storage nodes, based on the storage space capacity corresponding to the distributed storage node and the unit space capacity, calculating multiple storage space blocks corresponding to the distributed storage node; performing hash processing on the multiple storage space blocks to determine the hash slot corresponding to each storage space block.

[0059] It can be understood that each hash slot corresponds to a unit space capacity; at the same time, the multiple hash slots corresponding to each distributed storage node are only related to its own storage space capacity and unit space capacity, and have nothing to do with the storage space capacity corresponding to other distributed storage nodes. Therefore, when the distributed storage nodes are expanded, the mapping relationship between the original distributed storage nodes and the corresponding multiple hash slots will not change. Based on this, the stability of the system can be ensured when expanding or reducing the distributed storage node services.

[0060] In this embodiment, according to the specific storage capacity of each distributed storage node, the corresponding multiple hash slots can be determined, and then based on the multiple hash slots, reasonable allocation of files to be uploaded can be achieved, laying a foundation for improving the resource utilization of each distributed storage node.

[0061] It should be noted that, based on the file upload method described in the above embodiment, when downloading files, more efficient file downloading can be achieved based on the preset hash slot routing table and storage location index.

[0062] In one embodiment, the file download method further includes the following steps: Step 1: Generate a distributed storage node mount directory corresponding to each of the multiple distributed storage nodes.

[0063] Step 2: Mount each distributed storage node to its corresponding distributed storage node mount directory.

[0064] It should be noted that, based on the preset hash slot routing table described in the above embodiment, it is possible to determine the number of distributed storage nodes and the arrangement order of the distributed storage nodes in the preset hash slot routing table.

[0065] Exemplarily, based on the distributed storage nodes recorded in the preset hash slot routing table, distributed storage node mount directories corresponding to each of the multiple distributed storage nodes are generated at the gateway proxy layer; according to the arrangement order of the multiple distributed storage nodes in the preset hash slot routing table, the multiple distributed storage node mount directories are arranged in the same order; further, each distributed storage node is mounted to its corresponding distributed storage node mount directory through mounting.

[0066] In an exemplary embodiment, three distributed storage nodes are used as an example. Figure 4 As shown, three distributed storage node mount directories are generated on the gateway proxy layer, namely distributed storage node mount directory 1-distributed storage node mount directory 3; through mounting, distributed storage node 1 is mounted to distributed storage node mount directory 1, distributed storage node 2 is mounted to distributed storage node mount directory 2, and distributed storage node 3 is mounted to distributed storage node mount directory 3.

[0067] In this embodiment, a distributed storage node mount directory corresponding to each of the multiple distributed storage nodes is generated in the gateway proxy layer, and each distributed storage node is further mounted to the corresponding distributed storage node mount directory through the mounting method, so as to directly access the distributed storage node on the gateway proxy layer to download the corresponding file to be downloaded, thereby reducing the intermediate links in the file download process and improving the file download efficiency.

[0068] In one embodiment, according to the file identifier, accessing the distributed storage node corresponding to the first mount directory through the first mount directory, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client includes: If the file to be downloaded corresponding to the file identifier exists in the distributed storage node corresponding to the first mounting directory, the file to be downloaded is read from the first mounting directory and sent to the client.

[0069] Exemplarily, based on the file identifier, the distributed storage node corresponding to the first mount directory is accessed through the first mount directory to determine whether there is a file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory; if there is a file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory, the file to be downloaded is directly read from the first mount directory and sent to the client. If there is no file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory, it indicates that the file to be downloaded corresponding to the file identifier may be stored in other distributed storage nodes.

[0070] In this embodiment, based on the file identifier, it can be accurately determined whether the corresponding to-be-downloaded file exists in the distributed storage node corresponding to the first mount directory, which lays a foundation for accurately downloading the to-be-downloaded file required by the client.

[0071] In one embodiment, according to the file identifier, accessing the distributed storage node corresponding to the first mount directory through the first mount directory, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client, further includes the following steps: Step 1: If the corresponding distributed storage node in the first mount directory does not have a file to be downloaded corresponding to the file identifier, iteratively match multiple distributed storage node mount directories in a preset iteration order to determine a second mount directory.

[0072] The distributed storage node corresponding to the second mount directory stores the file to be downloaded. The second mount target is the distributed storage node mount directory corresponding to the distributed storage node storing the file to be downloaded corresponding to the file identifier.

[0073] Step 2: read the file to be downloaded corresponding to the file identifier from the second mounting directory and send it to the client.

[0074] Among them, the preset iteration order can be but is not limited to being related to the arrangement order of multiple distributed storage node mount directories, and needs to be set according to actual needs. No specific limitation is made here, but it is necessary to ensure that all distributed storage node mount directories can be traversed.

[0075] In an exemplary embodiment, a method for setting a preset iteration order may be: obtaining an arrangement order of multiple distributed storage node mount directories, and determining the preset iteration order according to the arrangement order of the multiple distributed storage node mount directories; for example, assuming that multiple distributed storage node mount directories are arranged in sequence from front to back, the preset iteration order may be an iteration order from front to back starting from the location of the second mount directory.

[0076] Exemplarily, based on the file identifier, the distributed storage node corresponding to the first mount directory is accessed through the first mount directory to determine whether there is a file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory; if there is a file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory, the file to be downloaded is directly read from the first mount directory and sent to the client. If there is no file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mount directory, iterative matching is performed on multiple distributed storage node mount directories in a preset iteration order to determine a second mount directory; further, the file to be downloaded corresponding to the file identifier is read from the second mount directory and sent to the client.

[0077] It should be noted that if after traversing all the distributed storage node mount directories based on the preset iteration order, it is determined that there is no second mount directory, it means that the current file to be downloaded is not stored in each distributed storage node. At this time, the gateway proxy layer will generate a download failure message and feedback to the client to remind the client that the current file to be downloaded cannot be successfully downloaded.

[0078] In this embodiment, when the corresponding distributed storage node in the first mount directory does not have the file to be downloaded corresponding to the file identifier, based on the preset iteration order, the second mount directory is determined by iteratively matching the mount directories of multiple distributed storage nodes, and then the file to be downloaded corresponding to the file identifier is read from the second mount directory and sent to the client. Based on this, a fault-tolerant mechanism based on the preset iteration order is implemented, ensuring that the required files to be downloaded can be successfully downloaded even if the storage location of some files is incorrect, thereby improving the reliability and accuracy of file downloading.

[0079] In a specific embodiment, see Figure 5 ,The file download method is applied to the gateway proxy layer, and specifically includes the following steps: Step S501: receiving a file download request sent by a client.

[0080] The file download request includes a storage location index and a file identifier corresponding to the file to be downloaded.

[0081] Step S502: According to the storage location index, a first mount directory corresponding to the storage location index is matched from multiple distributed storage node mount directories.

[0082] Step S503: Access the distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier.

[0083] Step S504: determine whether there is a file to be downloaded corresponding to the file identifier in the distributed storage node corresponding to the first mounting directory.

[0084] If so, execute step S505; otherwise, execute step S506.

[0085] Step S505: If the file to be downloaded corresponding to the file identifier exists in the distributed storage node corresponding to the first mounting directory, the file to be downloaded is read from the first mounting directory and sent to the client.

[0086] Step S506: If the corresponding distributed storage node in the first mount directory does not have the file to be downloaded corresponding to the file identifier, iterative matching is performed on multiple distributed storage node mount directories in a preset iteration order to determine a second mount directory.

[0087] Among them, the distributed storage node corresponding to the second mounting directory stores the files to be downloaded.

[0088] Step S507: read the file to be downloaded corresponding to the file identifier from the second mounting directory and send it to the client.

[0089] The last file download method, firstly, can quickly match the first mount directory corresponding to the storage location index from multiple distributed storage node mount directories according to the storage location index, effectively improving the search efficiency of the storage location of the file to be downloaded; further, according to the file identifier, the gateway proxy layer can directly access the distributed storage node corresponding to the first mount directory through the first mount directory, and then determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client, avoiding the problem of low file download efficiency caused by the long intermediate link of file download in the prior art; realizing file download through the gateway proxy layer can effectively shorten the intermediate link of the file download process and improve the file download efficiency. Secondly, for each file to be uploaded in the set of files to be uploaded, the target hash slot is determined by taking the modulo processing on the file upload order of the files to be uploaded, and according to the preset hash slot routing table and the target hash slot, multiple files to be uploaded can be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner. Based on this, the loads of multiple distributed storage nodes are balanced, and the utilization rate of the resources of each distributed storage node is improved. The load imbalance in the prior art when uploading large batches of files, that is, the overload of a single or a few storage nodes, is avoided, and the balance of file upload is effectively improved; based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes is achieved, which lays a foundation for improving the flexibility of the horizontal expansion of the distributed storage nodes. At the same time, the target hash slot corresponding to the file to be uploaded is saved as a storage location index, which lays a foundation for quickly locating the storage location corresponding to the file to be downloaded when downloading the file. On the third aspect, when the corresponding distributed storage node in the first mount directory does not have the file to be downloaded corresponding to the file identifier, based on the preset iteration order, the second mount directory is determined by iteratively matching the mount directories of multiple distributed storage nodes, and then the file to be downloaded corresponding to the file identifier is read from the second mount directory and sent to the client. Based on this, a fault-tolerant mechanism based on the preset iteration order is implemented, ensuring that even if the storage location of some files is incorrect, the files to be downloaded can be successfully downloaded, thereby improving the reliability and accuracy of file downloading.

[0090] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0091] Based on the same inventive concept, the embodiment of the present application also provides a file download device for implementing the file download method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file download device embodiments provided below can refer to the limitations on the file download method above, and will not be repeated here.

[0092] In an exemplary embodiment, Figure 6 As shown, a file downloading device is provided, including: a receiving module 601, a matching module 602 and a downloading module 603; The receiving module 601 is used to receive a file download request sent by a client; the file download request includes a storage location index and a file identifier corresponding to the file to be downloaded; A matching module 602 is used to match a first mount directory corresponding to the storage location index from multiple distributed storage node mount directories according to the storage location index; The download module 603 is used to access the distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client.

[0093] The above-mentioned file downloading device can quickly match the first mount directory corresponding to the storage location index from multiple distributed storage node mount directories according to the storage location index, thereby effectively improving the search efficiency of the storage location of the file to be downloaded; further, according to the file identifier, the gateway proxy layer can directly access the distributed storage node corresponding to the first mount directory through the first mount directory, and then determine the file to be downloaded pointed to by the file identifier, and send the file to be downloaded to the client, thereby avoiding the problem of low file download efficiency caused by the long intermediate link of file download in the prior art; realizing file download through the gateway proxy layer can effectively shorten the intermediate link of the file download process and improve the file download efficiency.

[0094] In one embodiment, the file downloading device further includes a file uploading module; File upload module for Obtain a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; For each file to be uploaded in the set of files to be uploaded, obtain the file upload order of the file to be uploaded in the set of files to be uploaded; Perform modulo processing on the order of file uploads to determine the target hash slot; According to the preset hash slot routing table and the target hash slot, a target storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the target storage node; The target hash slot is saved as the storage location index corresponding to the file to be uploaded.

[0095] In one embodiment, the file upload module is also used to Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the order of file uploads is modulo processed to determine the target hash slot.

[0096] In one embodiment, the file upload module is also used to Determine a plurality of hash slots corresponding to each distributed storage node according to the storage space capacity corresponding to each distributed storage node in the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each distributed storage node is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0097] In one embodiment, the file downloading device further includes a directory generation module; Catalog generation module for Generate distributed storage node mount directories corresponding to multiple distributed storage nodes; Mount each distributed storage node to its corresponding distributed storage node mount directory.

[0098] In one embodiment, the download module 603 is also used to If the file to be downloaded corresponding to the file identifier exists in the distributed storage node corresponding to the first mounting directory, the file to be downloaded is read from the first mounting directory and sent to the client.

[0099] In one embodiment, the download module 603 is also used to If the distributed storage node corresponding to the first mount directory does not have the file to be downloaded corresponding to the file identifier, iteratively matching the mount directories of multiple distributed storage nodes in a preset iteration order to determine a second mount directory; the distributed storage node corresponding to the second mount directory stores the file to be downloaded; The file to be downloaded corresponding to the file identifier is read from the second mounting directory and sent to the client.

[0100] Each module in the above-mentioned file downloading device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0101] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file download related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file download method is implemented.

[0102] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0103] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0104] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0105] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0106] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0108] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file downloading method, It is characterized in that Applied at the gateway proxy layer, the method includes: Receive a file download request sent by a client; the file download request includes a storage location index and a file identifier corresponding to the file to be downloaded; According to the storage location index, a first mount directory corresponding to the storage location index is matched from multiple distributed storage node mount directories; According to the file identifier, the distributed storage node corresponding to the first mount directory is accessed through the first mount directory to determine the file to be downloaded pointed to by the file identifier, and the file to be downloaded is sent to the client.

2. The method according to claim 1, It is characterized in that Before receiving the file download request sent by the client, the method further includes: Obtain a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; For each file to be uploaded in the set of files to be uploaded, obtaining a file uploading order of the file to be uploaded in the set of files to be uploaded; Performing a modulo process on the file upload order to determine the target hash slot; According to the preset hash slot routing table and the target hash slot, a target storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the target storage node; The target hash slot is saved as a storage location index corresponding to the file to be uploaded.

3. The method according to claim 2, It is characterized in that The performing modulo processing on the file upload order to determine the target hash slot includes: Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the file upload order is modulo processed to determine the target hash slot.

4. The method according to claim 3, It is characterized in that The obtaining the total number of hash slots corresponding to the plurality of distributed storage nodes includes: Determine a plurality of hash slots corresponding to each of the plurality of distributed storage nodes according to the storage space capacity corresponding to each of the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each of the distributed storage nodes is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

5. The method according to claim 1, It is characterized in that The method further comprises: Generate distributed storage node mount directories corresponding to multiple distributed storage nodes; Mount each of the distributed storage nodes to the corresponding distributed storage node mount directory.

6. The method according to claim 1, It is characterized in that The step of accessing the distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client includes: If the file to be downloaded corresponding to the file identifier exists in the distributed storage node corresponding to the first mounting directory, the file to be downloaded is read from the first mounting directory and sent to the client.

7. The method according to claim 1, It is characterized in that The method further comprises: accessing a distributed storage node corresponding to the first mount directory through the first mount directory according to the file identifier, determining the file to be downloaded pointed to by the file identifier, and sending the file to be downloaded to the client. If the distributed storage node corresponding to the first mount directory does not have the file to be downloaded corresponding to the file identifier, iteratively matching the mount directories of multiple distributed storage nodes in a preset iteration order to determine a second mount directory; the distributed storage node corresponding to the second mount directory stores the file to be downloaded; The file to be downloaded corresponding to the file identifier is read from the second mounting directory and sent to the client.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images