File uploading method and device, computer equipment and readable storage medium

Abstract

The invention relates to a file uploading method and device, computer equipment and a readable storage medium. Obtaining a to-be-uploaded file set; the to-be-uploaded file set comprises a plurality of to-be-uploaded files; for each to-be-uploaded file in the to-be-uploaded file set, if the to-be-uploaded file does not have a historical uploading record, acquiring a file uploading sequence of the to-be-uploaded file in the to-be-uploaded file set; performing modulo processing on the file uploading sequence, and determining a target hash slot; and according to a preset hash slot routing table and the target hash slot, matching a first storage node corresponding to the target hash slot from the plurality of distributed storage nodes, and uploading the to-be-uploaded file to the first storage node. On the basis, the loads of the multiple distributed storage nodes are balanced, the utilization rate of resources of all the distributed storage nodes is improved, the situation that in the prior art, when a large number of files are uploaded, the loads are unbalanced, namely, a single storage node or a few storage nodes are overloaded is avoided, and the file uploading balance is effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of data uploading, and in particular to a file uploading method, device, computer equipment and readable storage medium. Background Art

[0002] As network access channels continue to become richer and more diverse, the number of users is growing exponentially, placing higher demands on file upload services. Faced with a huge number of users and high-concurrency upload requests, the existing centralized cluster architecture faces many challenges, especially the problem of load imbalance; for example, some nodes may receive too many files and become overloaded, while other nodes may be idle.

[0003] There is no effective solution to the problem of load imbalance in the prior art when uploading large quantities of highly concurrent files. Summary of the invention

[0004] Based on this, it is necessary to provide a file uploading method, device, computer equipment and readable storage medium to address the above technical problems.

[0005] In a first aspect, the present application provides a method for uploading a file, the method comprising: Obtain a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; For each file to be uploaded in the set of files to be uploaded, if there is no historical upload record for the file to be uploaded, obtaining the file upload order of the file to be uploaded in the set of files to be uploaded; Performing a modulo process on the file upload order to determine the target hash slot; According to the preset hash slot routing table and the target hash slot, a first storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the first storage node.

[0006] In one embodiment, the method further comprises: Determine the storage location index of the file to be uploaded; According to the storage location index, matching a second storage node corresponding to the storage location index from a plurality of distributed storage nodes; If the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded.

[0007] In one embodiment, determining the storage location index of each file to be uploaded in the set of files to be uploaded includes: Obtain a file upload request corresponding to the file to be uploaded; the file upload request includes the name of the file to be uploaded; Mapping is performed on the file name to be uploaded to obtain a storage location index corresponding to the file to be uploaded.

[0008] In one embodiment, mapping the file name to be uploaded to obtain a storage location index corresponding to the file to be uploaded includes: Get the total number of hash slots corresponding to multiple distributed storage nodes; Performing data conversion processing on the file name to be uploaded to obtain a hash value corresponding to the file name to be uploaded; According to the total number of hash slots, the hash value is modulo processed to obtain the storage location index corresponding to the file to be uploaded.

[0009] In one embodiment, performing a modulo process on the file upload order to determine a target hash slot includes: Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the file upload order is modulo processed to determine the target hash slot.

[0010] In one embodiment, obtaining the total number of hash slots corresponding to the plurality of distributed storage nodes includes: Determine a plurality of hash slots corresponding to each of the plurality of distributed storage nodes according to the storage space capacity corresponding to each of the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each of the distributed storage nodes is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0011] In one embodiment, the method further comprises: If the storage location index is stored in the second storage node, obtaining the file transfer location corresponding to the file to be uploaded; Determining the remaining file segments to be uploaded in the file to be uploaded according to the file interruption position; The remaining file segments to be transmitted are uploaded to the second storage node.

[0012] In a second aspect, the present application further provides a file uploading device, the device comprising: An acquisition module is used to acquire a set of files to be uploaded; the set of files to be uploaded includes a plurality of files to be uploaded; A judgment module, configured to obtain, for each file to be uploaded in the set of files to be uploaded, a file upload order of the file to be uploaded in the set of files to be uploaded if there is no historical upload record for the file to be uploaded; A determination module, used for performing a modulo process on the file upload order to determine a target hash slot; The upload module is used to match a first storage node corresponding to the target hash slot from multiple distributed storage nodes according to a preset hash slot routing table and the target hash slot, and upload the file to be uploaded to the first storage node.

[0013] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the embodiments of the first aspect are implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in any one of the embodiments of the first aspect above are implemented.

[0015] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect above.

[0016] The above-mentioned file uploading method, device, computer equipment and readable storage medium; for each file to be uploaded in the set of files to be uploaded, when there is no historical upload record for the file to be uploaded, the target hash slot is determined by taking the modulus processing on the file upload order of the file to be uploaded, and according to the preset hash slot routing table and the target hash slot, multiple files to be uploaded can be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner. Based on this, not only the load of multiple distributed storage nodes is balanced, but also the utilization rate of the resources of each distributed storage node is improved, avoiding the load imbalance in the prior art when realizing large-scale high-concurrency file upload, that is, the overload of a single or a few storage nodes, and effectively improving the balance of file upload. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 An application environment diagram of a file uploading method in an embodiment; Figure 2 A schematic diagram of a process of uploading a file in one embodiment; Figure 3 A flowchart of a step of determining whether there is a historical upload record of a file to be uploaded in one embodiment; Figure 4 A flowchart of a file uploading method in another embodiment; Figure 5 A schematic diagram of a distributed file upload system in a specific embodiment; Figure 6 A flowchart of a method for uploading a file in a specific embodiment; Figure 7 It is a structural block diagram of a file uploading device in one embodiment; Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0020] The file upload method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and Internet of Things devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0021] In an exemplary embodiment, Figure 2 As shown, Figure 2 1 is a flowchart of a file upload method in an embodiment; this embodiment takes the method applied to a gateway proxy layer as an example, wherein the gateway proxy layer may include, but is not limited to, a proxy server; in this embodiment, the method includes the following steps: Step S201, obtaining a set of files to be uploaded.

[0022] The file set to be uploaded includes multiple files to be uploaded. It should be noted that the file set to be uploaded may be a large batch high-concurrency file set; each file to be uploaded in the file set to be uploaded comes from the client.

[0023] Step S202: for each file to be uploaded in the set of files to be uploaded, if there is no historical upload record for the file to be uploaded, the file upload order of the file to be uploaded in the set of files to be uploaded is obtained.

[0024] Among them, the historical upload record is used to determine whether the file to be uploaded is a new file that has not been uploaded; illustratively, if there is no historical upload record for the file to be uploaded, it indicates that the file to be uploaded is a new file and has not been uploaded; if there is a historical upload record for the file to be uploaded, it indicates that the file to be uploaded is an old file and has been partially or completely uploaded before.

[0025] Among them, the file upload order refers to the upload order number of the file to be uploaded in the file set to be uploaded; for example, assuming that the file set to be uploaded includes 1000 files to be uploaded, the file upload order includes 1-1000, among which the file upload order corresponding to the first file to be uploaded is 1, the file upload order corresponding to the second file to be uploaded is 2, and so on. Each file to be uploaded has a unique corresponding file upload order.

[0026] Step S203, performing a modulo process on the file upload order to determine the target hash slot.

[0027] The target hash slot is used to determine the storage location of the file to be uploaded.

[0028] Optionally, the method of performing modulo processing on the file upload order may be, but is not limited to, mapping changes to the file upload order to determine the corresponding target hash slot.

[0029] Step S204: According to the preset hash slot routing table and the target hash slot, a first storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the first storage node.

[0030] The distributed storage node may be a distributed server; wherein the distributed server includes a solid-state hard disk for providing storage space. It should be noted that the number of distributed storage nodes needs to be set according to the actual needs of file upload, and is not specifically limited here. The first storage node refers to a distributed storage node corresponding to the target hash slot among multiple distributed storage nodes.

[0031] The preset hash slot routing table includes a mapping relationship between hash slots and distributed storage nodes, and is used to quickly locate the first storage node corresponding to the target hash slot. It should be noted that the preset hash slot routing table needs to be set according to the number of multiple distributed storage nodes and the storage performance of each distributed storage node, which is not specifically limited here.

[0032] It should be noted that based on the preset hash slot routing table, when the distributed storage nodes need to be horizontally expanded, it is only necessary to add the mapping relationship between the expanded distributed storage nodes and their corresponding hash slots on the basis of the original preset hash slot routing table. There is no need to adjust the original mapping relationship. Based on this, the entire system can be easier to maintain and expand.

[0033] In an exemplary embodiment, the file upload distributed system includes a client, a gateway proxy layer, and three distributed storage nodes; the preset hash slot routing table is shown in Table 1, and there is a mapping relationship between distributed storage node 1 and hash slots 1-5; there is a mapping relationship between distributed storage node 2 and hash slots 6-11; there is a mapping relationship between distributed storage node 1 and hash slots 12-18. When expanding the distributed storage nodes, for example, expanding distributed storage node 4 in the system, at this time, it is only necessary to add a mapping relationship between distributed storage node 4 and hash slots 19-25 based on the original preset hash slot routing table.

[0034] Table 1 Preset hash slot routing table It should be noted that based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes can be achieved, and at the same time, the resources of each distributed storage node can be maximized and rationalized. Among them, the decoupling of the file to be uploaded and the distributed storage nodes means that even if a distributed storage node fails or needs maintenance, only the preset hash slot routing table needs to be updated, and there is no need to adjust the file upload logic; based on the above decoupling relationship, the architecture of the entire system can be made more flexible and easy to expand horizontally.

[0035] Exemplarily, for any file to be uploaded in the file set to be uploaded, determine whether there is a historical upload record for the file to be uploaded; if there is a historical upload record for the file to be uploaded, obtain the file upload order of the file to be uploaded in the file set to be uploaded; further, perform modulo processing on the file upload order to obtain the target hash slot; according to the preset hash slot routing table, the first storage node corresponding to the file to be uploaded can be accurately determined; for example, taking Table 1 as an example, assuming that the target hash slot corresponding to the file to be uploaded is 13, according to the preset hash slot routing table in Table 1, it can be determined that the first storage node corresponding to the target hash slot 13 is distributed storage node 3, and then the file to be uploaded can be uploaded to the distributed storage node 3.

[0036] In the present embodiment, for each file to be uploaded in the set of files to be uploaded, when there is a historical upload record for the file to be uploaded, the target hash slot is determined by taking the modulo processing on the file upload order of the file to be uploaded, and according to the preset hash slot routing table and the target hash slot, multiple files to be uploaded can be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner, thereby balancing the loads of multiple distributed storage nodes, improving the utilization rate of resources of each distributed storage node, avoiding the load imbalance in the prior art when uploading large quantities of files, that is, overloading of a single or a few storage nodes, and effectively improving the balance of file upload; at the same time, based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes is achieved, laying a foundation for improving the flexibility of horizontal expansion of distributed storage nodes.

[0037] In one embodiment, Figure 3 As shown, Figure 3 This is a flow chart of a step of determining whether there is a historical upload record of a file to be uploaded in one embodiment; the file uploading method further includes the following steps: Step S301, determining the storage location index of the file to be uploaded.

[0038] The storage location index is used to represent the physical storage location corresponding to the file to be uploaded; the physical storage location is a distributed storage node.

[0039] In an exemplary embodiment, the method for determining the storage location index of the file to be uploaded may be, but is not limited to: obtaining the file attribute corresponding to the file to be uploaded, performing hash processing on the file attribute to generate a corresponding hash value; performing modulo processing on the hash value to obtain the storage location index corresponding to the file to be uploaded. The file attribute may be, but is not limited to, the file name.

[0040] Step S302: According to the storage location index, a second storage node corresponding to the storage location index is matched from a plurality of distributed storage nodes.

[0041] The second storage node is a distributed storage node corresponding to the storage location index among the multiple distributed storage nodes.

[0042] It can be understood that there is a corresponding relationship between the storage location index and each distributed storage node; for example, when the storage location index is 1, it corresponds to distributed storage node 1, and when the storage location index is 2, it corresponds to distributed storage node 2.

[0043] Step S303: If the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded.

[0044] Exemplarily, the file attributes corresponding to the file to be uploaded are obtained, the file attributes are hashed to generate a corresponding hash value; the hash value is modulo processed to obtain a storage location index corresponding to the file to be uploaded; according to the storage location index, a second storage node corresponding to the storage location index is matched from multiple distributed storage nodes; it is determined whether the storage location index is stored in the second storage node; if the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded; if the storage location index is stored in the second storage node, it is determined that there is a historical upload record for the file to be uploaded.

[0045] It should be noted that when there is a historical upload record for a file to be uploaded, it indicates that the file to be uploaded has been partially or completely uploaded.

[0046] In this embodiment, based on the storage location index, the second storage node corresponding to the file to be uploaded can be quickly located, thereby improving the efficiency of storage location positioning. Furthermore, by judging whether the storage location index is stored in the second storage node, it is possible to accurately determine whether there is a historical upload record for the file to be uploaded, thereby laying a foundation for improving the load balancing of file uploads. At the same time, based on the storage location index, it is possible to lay a foundation for ensuring the consistency and integrity of files when resuming file uploads from breakpoints.

[0047] In one embodiment, determining the storage location index of each file to be uploaded in the set of files to be uploaded includes the following steps: Step 1: Obtain a file upload request corresponding to the file to be uploaded; the file upload request includes the name of the file to be uploaded.

[0048] The file upload request is sent by the client to the gateway proxy layer; the gateway proxy layer responds to the file upload request to achieve the file upload task of the file to be uploaded.

[0049] Step 2: Map the file name to be uploaded to obtain the storage location index corresponding to the file to be uploaded.

[0050] Among them, the mapping process aims to convert the file name to be uploaded into a storage location index, so as to efficiently manage and locate files in the file upload distributed system.

[0051] Preferably, step 2, mapping the file name to be uploaded to obtain the storage location index corresponding to the file to be uploaded, includes the following steps: Step 2.1, obtain the total number of hash slots corresponding to multiple distributed storage nodes.

[0052] Among them, the total number of hash slots corresponding to the distributed storage nodes is the sum of the numbers of hash slots corresponding to all distributed storage nodes; illustratively, taking Table 1 as an example, assuming that the file upload distributed system includes 3 distributed storage nodes, the amount of hash slot data corresponding to distributed storage node 1 is 5, the amount of hash slot data corresponding to distributed storage node 2 is 6, and the amount of hash slot data corresponding to distributed storage node 3 is 7. Further, by summing up the amount of hash slot data of each distributed storage node, it can be determined that the total number of hash slots is 18.

[0053] Step 2.2, perform data conversion processing on the file name to be uploaded to obtain the hash value corresponding to the file name to be uploaded.

[0054] The data conversion processing method may be hash processing, which is not specifically limited here.

[0055] Step 2.3, according to the total number of hash slots, perform modulo processing on the hash value to obtain the storage location index corresponding to the file to be uploaded.

[0056] Exemplarily, assume that the total number of hash slots corresponding to multiple distributed storage nodes is 100; the name of the file to be uploaded is "fileName.txt"; by calling the hashCode method of the String class, the data conversion processing is performed on the file name to be uploaded "fileName.txt" to obtain the hash value corresponding to the file name to be uploaded, for example, "int hashCode =fileName.hashCode()"; further, according to the total number of hash slots 100, the hash value is modulo processed to obtain the storage location index corresponding to the file to be uploaded; for example, "int index = hashCode %arraySize", where hashCode represents the hash value corresponding to the file name to be uploaded; arraySize represents the total number of hash slots.

[0057] In this embodiment, by performing data conversion processing on the file name to be uploaded, a hash value corresponding to the file name to be uploaded is obtained; further, according to the total number of hash slots, the hash value is modulo processed to obtain the storage location index corresponding to the file to be uploaded, based on which the upload status of the file to be uploaded can be accurately monitored, laying a foundation for load balancing of file uploads, and at the same time laying a foundation for ensuring the consistency and integrity of files when resuming file uploads from breakpoints.

[0058] In one embodiment, performing a modulo process on the file upload order to determine the target hash slot includes the following steps: Step 1: Get the total number of hash slots corresponding to multiple distributed storage nodes.

[0059] Step 2: According to the total number of hash slots, perform modulo processing on the file upload order to determine the target hash slot.

[0060] Exemplarily, according to the total number of hash slots corresponding to multiple distributed storage nodes, the order of file uploads is modulo processed, and the specific process of determining the target hash slot is as follows: Wherein, N represents the file upload order of the to-be-uploaded files in the to-be-uploaded file set; X represents the total number of hash slots corresponding to multiple distributed storage nodes; and Y represents the target hash slot.

[0061] In this embodiment, based on the total number of hash slots, the order of file uploads is modulo processed to determine the target hash slot, and then based on the target hash slot, a foundation can be laid for achieving load balancing of file uploads.

[0062] In one embodiment, obtaining the total number of hash slots corresponding to multiple distributed storage nodes includes the following steps: Step 1: Determine a plurality of hash slots corresponding to each distributed storage node according to the storage space capacity corresponding to each distributed storage node in the plurality of distributed storage nodes.

[0063] The storage space capacity refers to the effective storage space size corresponding to the distributed storage node; for example, the storage space size of the solid-state disk in the distributed server.

[0064] Step 2: sum the number of hash slots corresponding to each distributed storage node to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0065] In an exemplary embodiment, a method for determining multiple hash slots corresponding to each distributed storage node according to the storage space capacity corresponding to each distributed storage node among multiple distributed storage nodes may be: for each distributed storage node among multiple distributed storage nodes, based on the storage space capacity corresponding to the distributed storage node and the unit space capacity, calculating multiple storage space blocks corresponding to the distributed storage node; performing hash processing on the multiple storage space blocks to determine the hash slot corresponding to each storage space block.

[0066] It can be understood that each hash slot corresponds to a unit space capacity; at the same time, the multiple hash slots corresponding to each distributed storage node are only related to its own storage space capacity and unit space capacity, and have nothing to do with the storage space capacity corresponding to other distributed storage nodes. Therefore, when the distributed storage nodes are expanded, the mapping relationship between the original distributed storage nodes and the corresponding multiple hash slots will not change. Based on this, the stability of the system can be ensured when expanding or reducing the distributed storage node services.

[0067] In this embodiment, according to the specific storage capacity of each distributed storage node, the corresponding multiple hash slots can be determined, and then based on the multiple hash slots, the reasonable allocation of files to be uploaded can be achieved, laying the foundation for improving the resource utilization of each distributed storage node and the balancing of file upload load.

[0068] In one embodiment, Figure 4 As shown, Figure 4 Schematic diagram of a flow chart of a file uploading method in another embodiment; the file uploading method further includes the following steps: Step S401: If the second storage node stores a storage location index, then obtain the file transfer location corresponding to the file to be uploaded.

[0069] It should be noted that the second storage node stores a storage location index, indicating that the file to be uploaded has been partially or completely uploaded in the second storage node; at this time, according to the file interruption location, it can be determined whether the file to be uploaded has been completely uploaded.

[0070] The file transfer interruption position refers to the data position where the file to be uploaded was last successfully transferred.

[0071] It should be noted that before the file to be uploaded is uploaded completely, the corresponding write position of the file to be uploaded needs to be recorded to ensure that the corresponding file transfer position can be determined when the file is interrupted, so that the remaining file fragments in the file to be uploaded can continue to be uploaded from the file transfer position.

[0072] In an exemplary embodiment, the method for obtaining the file discontinuation position corresponding to the file to be uploaded may be: obtaining the last written position of the file to be uploaded; and using the last written position as the file discontinuation position.

[0073] Step S401, determining the remaining file segments to be uploaded according to the file interruption position.

[0074] The remaining file segments to be uploaded refer to the file segments that have not been uploaded in the files to be uploaded.

[0075] Step S403: Upload the remaining file segments to be transmitted to the second storage node.

[0076] It should be noted that during the upload process of the file to be uploaded, there may be a situation where the transmission is interrupted due to network interruption, system failure or other reasons, so it is necessary to resume the file to be uploaded to ensure the integrity and consistency of the file to be uploaded. In the prior art, when resuming the file to be uploaded, if the first file upload is performed on the distributed storage node A, the second file upload may be performed again from the beginning on the distributed storage node B. This will not only waste the upload time due to re-uploading the file, but also cause a waste of resources due to the large number of incomplete file fragments distributed on the distributed storage node A, occupying the storage resources of the distributed storage node A.

[0077] The present embodiment is based on the storage location index. By being able to accurately determine whether the storage location index is stored in the second storage node, if the storage location index is stored in the second storage node, then by obtaining the file interruption position corresponding to the file to be uploaded, the remaining file fragments to be uploaded can be resumed, which can effectively avoid the problems of inconsistent file uploads, waste of resources, and time consumption in the above-mentioned prior art.

[0078] Specifically, determine the storage location index of the file to be uploaded; match a second storage node corresponding to the storage location index from multiple distributed storage nodes based on the storage location index; determine whether the storage location index is stored in the second storage node, if the storage location index is not saved in the second storage node, determine that there is no historical upload record for the file to be uploaded; if the storage location index is stored in the second storage node, obtain the file discontinuity position corresponding to the file to be uploaded; determine the remaining file fragments to be transmitted in the file to be uploaded based on the file discontinuity position; upload the remaining file fragments to be transmitted to the second storage node.

[0079] In the present embodiment, based on the storage location index and the file interruption position, it is possible to determine whether the file to be uploaded has been partially or completely uploaded in the second storage node; at the same time, based on the file interruption position, the remaining file fragments to be transmitted in the file to be uploaded can be determined, and then the remaining file fragments to be transmitted in the file to be uploaded can be uploaded to the second storage node. Based on this, not only the consistency and integrity of the file to be uploaded can be ensured, but also the waste of resources and low file upload efficiency caused by repeated uploading of the uploaded file fragments in the file to be uploaded can be avoided.

[0080] In a specific embodiment, a file upload distributed system such as Figure 5 As shown in Figure 1, the file upload distributed system includes a client, a gateway proxy layer, and multiple distributed storage nodes. Figure 6 , taking the application of the file upload method in the gateway proxy layer in the file upload distributed system as an example for explanation, specifically including the following steps: Step S601: receiving a file set to be uploaded sent by a client.

[0081] The file set to be uploaded includes multiple files to be uploaded.

[0082] Step S602: for each file to be uploaded in the file set to be uploaded, determine a storage location index of the file to be uploaded.

[0083] Step S603: According to the storage location index, a second storage node corresponding to the storage location index is matched from a plurality of distributed storage nodes.

[0084] Step S604: determine whether the second storage node stores a storage location index.

[0085] If so, execute step S609; otherwise, execute step S605.

[0086] Step S605: If the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded.

[0087] Step S606: Obtain the file upload order of the files to be uploaded in the set of files to be uploaded.

[0088] Step S607, performing a modulo process on the file upload order to determine the target hash slot.

[0089] Step S608: According to the preset hash slot routing table and the target hash slot, a first storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the first storage node.

[0090] Step S609: If the second storage node stores a storage location index, then obtain the file transfer location corresponding to the file to be uploaded.

[0091] Step S610, determining the remaining file segments to be uploaded according to the file interruption position.

[0092] Step S611, uploading the remaining file segments to be transmitted to the second storage node.

[0093] In the above-mentioned file uploading method, on the first aspect, for each file to be uploaded in the set of files to be uploaded, when there is no historical record of the file to be uploaded, the target hash slot is determined by taking the modulo processing on the file upload order of the file to be uploaded, and according to the preset hash slot routing table and the target hash slot, multiple files to be uploaded can be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner, thereby balancing the loads of multiple distributed storage nodes, improving the utilization rate of the resources of each distributed storage node, avoiding the load imbalance in the prior art when uploading large quantities of files, that is, overloading of a single or a few storage nodes, and effectively improving the balance of file uploading; at the same time, based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes is achieved, laying a foundation for improving the flexibility of horizontal expansion of distributed storage nodes. Secondly, based on the storage location index and the file interruption position, it is possible to determine whether the file to be uploaded has been partially or completely uploaded in the second storage node; at the same time, based on the file interruption position, the remaining file fragments to be transmitted in the file to be uploaded can be determined, and then the remaining file fragments to be transmitted in the file to be uploaded can be uploaded to the second storage node. Based on this, not only the consistency and integrity of the file to be uploaded can be ensured, but also the waste of resources caused by repeated uploading of the uploaded file fragments in the file to be uploaded and the problem of low file upload efficiency can be avoided.

[0094] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiment of the present application also provides a file uploading device for implementing the file uploading method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more file uploading device embodiments provided below can refer to the limitations on the file uploading method above, and will not be repeated here.

[0096] In an exemplary embodiment, Figure 7As shown, a file uploading device is provided, including: an acquisition module 701, a judgment module 702, a determination module 703 and an uploading module 704; The acquisition module 701 acquires a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; The judgment module 702 is used for obtaining the file upload order of each file to be uploaded in the file set to be uploaded if there is no historical upload record for the file to be uploaded; A determination module 703 is used to perform a modulo process on the file upload order to determine a target hash slot; The upload module 704 is used to match a first storage node corresponding to the target hash slot from multiple distributed storage nodes according to a preset hash slot routing table and a target hash slot, and upload the file to be uploaded to the first storage node.

[0097] The above-mentioned file uploading device, for each file to be uploaded in the set of files to be uploaded, determines the target hash slot by taking the modulo processing on the file upload order of the files to be uploaded, and can upload multiple files to be uploaded to the first storage node corresponding to the target hash slot in a more balanced manner according to the preset hash slot routing table and the target hash slot. Based on this, the loads of multiple distributed storage nodes are balanced, the utilization rate of the resources of each distributed storage node is improved, and the load imbalance when realizing large-scale file uploading, that is, the overload of a single or a few storage nodes in the prior art is avoided, and the balance of file uploading is effectively improved; at the same time, based on the target hash slot of the file to be uploaded and the preset hash slot routing table, the decoupling of the file to be uploaded and the distributed storage nodes is achieved, laying a foundation for improving the flexibility of horizontal expansion of distributed storage nodes.

[0098] In one embodiment, the determination module 702 is further configured to: Determine the storage location index of the file to be uploaded; According to the storage location index, matching a second storage node corresponding to the storage location index from a plurality of distributed storage nodes; If the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded.

[0099] In one embodiment, the determination module 702 is further configured to: Obtain a file upload request corresponding to the file to be uploaded; the file upload request includes the name of the file to be uploaded; Map the file name to be uploaded to obtain the storage location index corresponding to the file to be uploaded.

[0100] In one embodiment, the determination module 702 is further configured to: Get the total number of hash slots corresponding to multiple distributed storage nodes; Perform data conversion on the file name to be uploaded to obtain the hash value corresponding to the file name to be uploaded; According to the total number of hash slots, the hash value is modulo processed to obtain the storage location index corresponding to the file to be uploaded.

[0101] In one embodiment, the determination module 703 is further configured to: Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the order of file uploads is modulo processed to determine the target hash slot.

[0102] In one embodiment, the determination module 703 is further configured to: Determine a plurality of hash slots corresponding to each distributed storage node according to the storage space capacity corresponding to each distributed storage node in the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each distributed storage node is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

[0103] In one embodiment, the upload module 704 is also used to If the storage location index is stored in the second storage node, the file transfer location corresponding to the file to be uploaded is obtained; According to the location where the file was interrupted, the remaining file segments to be uploaded are determined; Upload the remaining file segments to be transferred to the second storage node.

[0104] Each module in the above-mentioned file uploading device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module above.

[0105] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file upload related data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a file upload method is implemented.

[0106] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0107] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0108] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0110] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0111] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0112] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A file uploading method, characterized in that: The method comprises: Obtain a set of files to be uploaded; the set of files to be uploaded includes multiple files to be uploaded; For each file to be uploaded in the set of files to be uploaded, if there is no historical upload record for the file to be uploaded, obtaining the file upload order of the file to be uploaded in the set of files to be uploaded; Performing a modulo process on the file upload order to determine the target hash slot; According to the preset hash slot routing table and the target hash slot, a first storage node corresponding to the target hash slot is matched from multiple distributed storage nodes, and the file to be uploaded is uploaded to the first storage node.

2. The method according to claim 1, characterized in that The method further comprises: Determine the storage location index of the file to be uploaded; According to the storage location index, matching a second storage node corresponding to the storage location index from a plurality of distributed storage nodes; If the storage location index is not saved in the second storage node, it is determined that there is no historical upload record for the file to be uploaded.

3. The method according to claim 1, characterized in that: The step of determining the storage location index of each file to be uploaded in the set of files to be uploaded includes: Obtain a file upload request corresponding to the file to be uploaded; the file upload request includes the name of the file to be uploaded; Mapping is performed on the file name to be uploaded to obtain a storage location index corresponding to the file to be uploaded.

4. The method according to claim 3, characterized in that The mapping process is performed on the file name to be uploaded to obtain a storage location index corresponding to the file to be uploaded, including: Get the total number of hash slots corresponding to multiple distributed storage nodes; Performing data conversion processing on the file name to be uploaded to obtain a hash value corresponding to the file name to be uploaded; According to the total number of hash slots, the hash value is modulo processed to obtain the storage location index corresponding to the file to be uploaded.

5. The method according to claim 1, characterized in that The performing modulo processing on the file upload order to determine the target hash slot includes: Get the total number of hash slots corresponding to multiple distributed storage nodes; According to the total number of hash slots, the file upload order is modulo processed to determine the target hash slot.

6. The method according to claim 3 or claim 4, characterized in that: Get the total number of hash slots corresponding to multiple distributed storage nodes, including: Determine a plurality of hash slots corresponding to each of the plurality of distributed storage nodes according to the storage space capacity corresponding to each of the plurality of distributed storage nodes; The number of multiple hash slots corresponding to each of the distributed storage nodes is summed to obtain the total number of hash slots corresponding to the multiple distributed storage nodes.

7. The method according to claim 2, characterized in that The method further comprises: If the storage location index is stored in the second storage node, obtaining the file transfer location corresponding to the file to be uploaded; Determining the remaining file segments to be uploaded in the file to be uploaded according to the file interruption position; The remaining file segments to be transmitted are uploaded to the second storage node.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images