A file upload method, apparatus, device and medium

Abstract

This invention discloses a file upload method, apparatus, device, and medium. The method includes: obtaining network bandwidth rate, storage space of a target file, and the number of available data nodes; calculating the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes; splitting the target file according to the network bandwidth rate to obtain at least one target sub-file; the target file includes at least one target sub-file; allocating each target sub-file to each available data node according to the storage capacity of each available data node, obtaining a mapping relationship between each target sub-file and each available data node; and uploading each target sub-file to each available data node according to the mapping relationship. Embodiments of this invention can improve the efficiency of file transfer.

Description

Technical Field

[0001] This invention relates to the field of distributed technology, and in particular to a file upload method, apparatus, device, and medium. Background Technology

[0002] In a distributed system, data files need to be uploaded from the client to the various data nodes of the server.

[0003] Currently, existing technologies involve randomly uploading data files to various data nodes on a server.

[0004] However, the above method may cause some data nodes to be under excessive storage pressure, while other nodes are under light load. Furthermore, due to the random allocation, when data files need to be downloaded, the client may find it difficult to accurately track the storage location of each sub-file. Summary of the Invention

[0005] This invention provides a file upload method, apparatus, device, and medium. Embodiments of this invention can improve the efficiency of file transmission.

[0006] In a first aspect, embodiments of the present invention provide a file upload method, the method comprising:

[0007] Obtain the network bandwidth rate, the storage space of the target file, and the number of available data nodes;

[0008] Calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes;

[0009] Based on the network bandwidth rate, the target file is split into at least one target sub-file; the target file includes at least one target sub-file.

[0010] Based on the storage capacity of each available data node, each target sub-file is allocated to each available data node, thus obtaining the mapping relationship between each target sub-file and each available data node;

[0011] Based on the mapping relationship, each target sub-file is uploaded to each available data node.

[0012] Secondly, embodiments of the present invention also provide a file uploading device, the device comprising:

[0013] The data acquisition module is used to obtain network bandwidth rate, target file storage space and number of available data nodes;

[0014] The storage capacity calculation module is used to calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes.

[0015] The target file splitting module is used to split the target file according to the network bandwidth rate to obtain at least one target sub-file; the target file includes at least one target sub-file.

[0016] The target sub-file allocation module is used to allocate each target sub-file to each available data node according to the storage capacity of each available data node, and obtain the mapping relationship between each target sub-file and each available data node;

[0017] The target sub-file upload module is used to upload each target sub-file to each available data node according to the mapping relationship.

[0018] Thirdly, embodiments of the present invention also provide a file upload device, the file upload device comprising:

[0019] At least one processor; and

[0020] A memory that is communicatively connected to at least one processor; wherein,

[0021] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the file upload method of any embodiment of the present invention.

[0022] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions, which are used to cause a processor to execute and implement the file upload method of any embodiment of the present invention.

[0023] The technical solution of this invention provides basic data support for subsequent steps by acquiring network bandwidth rate, target file size, and number of available data nodes. By calculating the storage capacity of each data node based on the storage space of the target file and the number of nodes, a balanced distribution of the target sub-file among the data nodes can be achieved, avoiding overload of a single node. By splitting the file according to network bandwidth, the size of the target sub-file can be matched with the actual network transmission capacity, thereby improving transmission efficiency and reducing the risk of network congestion during transmission. By allocating the target sub-file based on the storage capacity of the data nodes, distributed storage that better matches the load capacity of each node can be achieved. By uploading the target sub-file according to the mapping relationship, a one-to-one correspondence between the target sub-file and the available data nodes can be achieved, ensuring the correct distribution of the file. This invention solves the technical problem that random storage causes excessive storage pressure on a single data node, making it difficult to accurately track the storage location of each sub-file. This invention can improve the efficiency of file transmission.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a file upload method provided in an embodiment of the present invention;

[0027] Figure 2 A flowchart of a file upload method provided in an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of the structure of a file upload device provided in an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of a file upload device provided in an embodiment of the present invention. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] In the technical solutions of this invention, the acquisition, storage, and application of network bandwidth rate, target file storage space, and number of available data nodes all comply with relevant laws and regulations and do not violate public order and good morals.

[0033] Figure 1 This is a flowchart illustrating a file upload method provided in an embodiment of the present invention. This embodiment is applicable to situations where a file is uploaded from a client to a data node on a server. The method can be executed by a file upload device, which can be implemented in hardware and / or software.

[0034] See Figure 1 The file upload methods shown include:

[0035] S101. Obtain the network bandwidth rate, the storage space of the target file, and the number of available data nodes.

[0036] Network bandwidth rate refers to the amount of data a network can transmit per unit of time. During data transmission, network bandwidth rate determines the number of bytes that can be sent or received per second. By obtaining the network bandwidth rate between the client and each available data node, network congestion and throughput can be detected, thereby determining the appropriate data packet size. Network bandwidth rate is used to guide the granularity of target file splitting, avoiding excessively large sub-files that cause transmission delays, and excessively small sub-files that cause scheduling overhead.

[0037] The target file refers to the data file that needs to be uploaded to the data nodes of the distributed system. The target file is usually a complete data set, and after uploading, it must support integrity verification and reversibility.

[0038] In this context, an available data node refers to a server-side data node that is in a connected state and can be selected as the storage location for the target sub-file. During file upload, all registered data nodes are traversed, and their connection status is checked to determine which data nodes are in an available state. These data nodes are then included in the sub-file allocation scope and marked as available data nodes.

[0039] S102. Calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes.

[0040] Storage capacity can refer to the amount of space available in each data node that can be used to store the target sub-file.

[0041] S103. Based on the network bandwidth rate, split the target file to obtain at least one target sub-file; the target file includes at least one target sub-file.

[0042] Splitting refers to dividing a complete target file into multiple independent data fragments according to preset or dynamically adjusted rules. These fragments are then uploaded to different data nodes, achieving distributed storage. Splitting can be based on bandwidth-adaptive rules, determining the size of the resulting sub-files based on bandwidth load. During the splitting process, metadata (such as indexes, offsets, and sequence numbers) of the sub-files must be preserved; this metadata supports subsequent target file reconstruction.

[0043] In this context, a target sub-file refers to a data unit that is split from a target file and has independent upload capabilities. Each target sub-file contains a portion of the target file's content, along with the necessary metadata required for reassembly. For example, a 500MB image compressed file might be split into five target sub-files, each 100MB in size.

[0044] S104. Based on the storage capacity of each available data node, allocate each target sub-file to each available data node to obtain the mapping relationship between each target sub-file and each available data node.

[0045] The mapping relationship refers to the association between target sub-files and available data nodes. The mapping relationship records the storage target location of each target sub-file, i.e., which available data node the target sub-file needs to be uploaded to.

[0046] S105. Based on the mapping relationship, upload each target sub-file to each available data node.

[0047] Specifically, based on the data node corresponding to each target sub-file recorded in the mapping relationship, a data transmission channel is established with the corresponding available data node. The transmission channel can be based on the TCP / IP protocol, or implemented using HTTP, FTP, or Socket connections to ensure the stability and security of the network connection. Before the formal transmission, a handshake request is sent to each available data node to confirm that the corresponding node is in a successfully connected state and in a state that can receive data. If a node responds abnormally, a fault tolerance mechanism can be triggered to reallocate the target sub-file to other available data nodes. After each target sub-file is uploaded, it can perform a verification operation in collaboration with the corresponding data node, such as calculating and comparing the MD5 value or hash digest, to confirm that the file content has not been damaged or lost during transmission. After the upload is completed, the upload status and data node information of each target sub-file are recorded, the mapping relationship is updated, and the corresponding sub-file is marked as successfully written to its target node for subsequent indexing, downloading, or recovery operations.

[0048] As can be seen, in this embodiment, by obtaining the network bandwidth rate, target file size, and number of available data nodes, basic data support can be provided for subsequent steps. By calculating the storage capacity of each data node based on the storage space of the target file and the number of nodes, a balanced distribution of the target sub-file among the data nodes can be achieved, avoiding overload of a single node. By splitting the file according to the network bandwidth, the size of the target sub-file can be matched with the actual network transmission capacity, thereby improving transmission efficiency and reducing the risk of network congestion during transmission. By allocating the target sub-file based on the storage capacity of the data nodes, distributed storage that is more in line with the load capacity of each node can be achieved. By uploading the target sub-file according to the mapping relationship, a one-to-one correspondence between the target sub-file and the available data nodes can be achieved, ensuring the correct distribution of the file. This invention solves the technical problem that the storage pressure on a single data node is too high due to random storage, making it difficult to accurately track the storage location of each sub-file. This embodiment of the invention can improve the efficiency of file transmission.

[0049] In an optional embodiment, Figure 2 The flowchart of a file upload method provided in this embodiment of the invention refines the process of "allocating each target sub-file to each available data node according to the storage capacity of each available data node, and obtaining the mapping relationship between each target sub-file and each available data node" into "determining the number of target sub-files that can be allocated in each available data node according to the storage capacity of each available data node; allocating each target sub-file to each available data node according to the number of target sub-files that can be allocated in each available data node, and obtaining the mapping relationship between each target sub-file and each available data node", thereby improving the file upload operation.

[0050] It should be noted that for parts not described in detail in the embodiments of the present invention, please refer to the descriptions in other embodiments.

[0051] See Figure 2 The file upload methods shown include:

[0052] S201. Obtain the network bandwidth rate, the storage space of the target file, and the number of available data nodes.

[0053] S202. Calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes.

[0054] S203. Based on the network bandwidth rate, split the target file to obtain at least one target sub-file; the target file includes at least one target sub-file.

[0055] S204. Determine the number of target sub-files that can be allocated in each available data node based on the storage capacity in each available data node.

[0056] Specifically, based on the size information of the target sub-file and combined with the current storage capacity of each available data node, the number of target sub-files that can be allocated in each available data node is calculated. Preferably, in order to avoid write overflow of available data nodes, when the storage capacity is close to the critical state, appropriate adjustments can be made with reference to parameters such as node response latency and transmission rate.

[0057] S205. Based on the number of allocable target sub-files in each available data node, allocate each target sub-file to each available data node to obtain the mapping relationship between each target sub-file and each available data node.

[0058] Specifically, each target sub-file is traversed, and the target sub-files are allocated to appropriate available data nodes in sequence according to strategies such as minimum load priority, round-robin balancing, or network latency optimization. Based on the allocation results, a mapping relationship is generated. The mapping relationship records the address of the available data node, sub-file number, and upload order index that each target sub-file should correspond to during its upload stage, which is used for scheduling and calling in subsequent upload steps.

[0059] S206. Based on the mapping relationship, upload each target sub-file to each available data node.

[0060] As can be seen, in this embodiment, by pre-calculating the number of allocatable target sub-files based on the storage capacity of available data nodes, storage capacity can be detected before resource allocation, improving the targeting and adaptability of data distribution; by distributing target sub-files to each available data node according to the determined allocation number, more accurate and efficient data upload path planning can be achieved; and by generating mapping relationships, structured index support can be provided for subsequent file recovery, download, or redistribution, facilitating operation and location.

[0061] In some embodiments, the number of available data nodes can be obtained in the following ways:

[0062] Obtain the connection status of each data node;

[0063] When the connection status of a data node is successful, the data node is marked as an available data node;

[0064] The number of available data nodes is obtained by statistically analyzing each available data node.

[0065] The connection status refers to the connection status between a data node and a client, indicating whether the data node is in a normal communication state. Only data nodes with a successful connection status are considered usable for uploading, downloading, or storage scheduling, ensuring the stability and real-time performance of data operations. By identifying nodes with abnormal connection statuses (such as disconnected, failed, or timed-out states), faulty nodes can be automatically avoided in task scheduling, improving the overall system robustness.

[0066] Specifically, if a stable TCP connection is established between a data node A and a client, the system marks it as a successfully connected node, and this node can participate in the file allocation task. If node B is in an unconnected state, even if its storage capacity is sufficient, the system will not allocate the target sub-file to this node. By counting the number of available data nodes with a successfully connected state, the total number of available data nodes can be obtained.

[0067] As can be seen, in this embodiment, by acquiring the connection status of each data node, the current communication status of each data node can be monitored in real time, thus providing basic information for subsequent screening of available data nodes; by marking data nodes as available data nodes only when the connection status is successful, the nodes that can actually participate in data operations can be accurately identified, thereby improving the success rate of data allocation or transmission; by statistically analyzing the available data nodes, the total number of usable data nodes in the current system can be obtained, providing an accurate quantitative basis for subsequent data task scheduling and resource allocation.

[0068] In some embodiments, the connection state includes at least one of the following: connection successful state, connecting state, and connection failed state.

[0069] A successful connection status indicates that a network connection has been established between the data node and the client, and they are in a communicable state. A successful connection status signifies that the data node can normally participate in operations such as data interaction, task scheduling, or file transfer.

[0070] The "connecting" state can refer to a data node being in the middle of the connection establishment process, before all connection procedures are completed. When a client attempts to establish a connection with a data node, the connection process involves multiple steps, including initiating a connection, waiting for a response, authentication, and handshake protocol. Before all steps are completed, the node is in an uncertain state of "connecting".

[0071] A connection failure state refers to a state where the client is unable to establish a usable communication link with the data node. Errors during the connection process (such as authentication failure, response timeout, or host unreachable errors) cause the connection to be interrupted. Poor network quality, node offline status, and closed ports can all lead to connection attempts failing. Once the connection retry threshold is exceeded or the network anomaly is not resolved, the data node will be marked as a connection failure state.

[0072] As can be seen, in this embodiment, by setting the connection status to include connection success, connection in progress, and connection failure, a comprehensive judgment on the current connection status of data nodes can be achieved, thereby improving the accuracy and real-time performance of data node status management, providing a more reliable basis for node selection for subsequent data upload tasks, and thus improving the stability and efficiency of the file upload process.

[0073] In some embodiments, according to the mapping relationship, each target sub-file is uploaded to each available data node, including:

[0074] Based on the mapping relationship, obtain the node identifier of each available data node;

[0075] Based on the node identifier, each target sub-file is uploaded to each available data node.

[0076] In this context, a node identifier refers to an information identifier that uniquely identifies a specific available data node. Node identifiers can be used to accurately and reliably identify a specific data node in a distributed environment. Methods of node identification include, but are not limited to, the data node's IP address, hostname, MAC address, or unique node code.

[0077] As can be seen, in this embodiment, by obtaining the node identifiers of each available data node based on the mapping relationship, the upload target node of the target sub-file can be quickly located, improving the efficiency of determining the upload path; by uploading the target sub-file to each available data node according to the node identifier, the sub-file can be accurately distributed, ensuring the correct storage of the target file in the distributed system and the consistency of the mapping relationship.

[0078] In some embodiments, after splitting the target file according to the network bandwidth rate to obtain at least one target sub-file, the method further includes:

[0079] Based on the mapping relationship, the index information of the target file is generated; the index information records the concatenation order of each target sub-file in the target file.

[0080] The index information refers to data that identifies and records the concatenation order of the various target sub-files within the target file. Index information typically exists in structured data format and is used to ensure that the target file can still be accurately reconstructed as the original target file after it has been split and distributed during upload.

[0081] As can be seen, in this embodiment, by generating index information based on the mapping relationship, the structural relationship between the sub-file and the original file can be effectively tracked, thereby providing a structured reference for file reconstruction; by recording the splicing order of each target sub-file in the target file, accurate reassembly after file upload can be achieved, ensuring that the file can still be restored to the correct original file content after being stored in segments.

[0082] In some embodiments, it also includes:

[0083] Get download request;

[0084] Based on the download request, obtain the mapping relationship between each target sub-file and each available data node;

[0085] Based on the mapping relationship, obtain the identifier of each available data node;

[0086] Based on the identifier, download each target sub-file from each available data node;

[0087] Based on the target subfiles and index information, generate and transmit the target file.

[0088] Specifically, the system receives download requests from user terminals, which carry identification information of the target file, such as file ID or file name, to uniquely identify the target file to be downloaded. Based on the target file identifier in the download request, the system retrieves the mapping information between each target sub-file and its corresponding available data node from a pre-stored mapping relationship generated during the upload phase. This mapping relationship records the location of each available data node where the target file is split into sub-files. Based on the correspondence recorded in the mapping relationship, the system extracts the node identifier information of each available data node. The node identifier may include the node's network address, node number, and port information, used to clearly identify the target node where each target sub-file is located. Based on the node identifier of each available data node, the system sends data access requests to the corresponding data node to download the corresponding target sub-file. To improve download efficiency, target sub-files can be downloaded concurrently from multiple data nodes simultaneously. After all target sub-files have been downloaded, based on the index information generated during the upload phase, the target sub-files are concatenated and reassembled in a predetermined order to restore the original target file. After concatenation, the restored target file is transmitted to the client that initiated the download request.

[0089] As can be seen, in this embodiment, by obtaining download requests, the system can respond to user or application access needs, thereby triggering the corresponding file download process; by obtaining mapping relationships based on download requests, the storage location of the required sub-files can be accurately located, thereby improving the efficiency and accuracy of file downloads; by obtaining data node identifiers based on mapping relationships, the download path of the target sub-file can be clearly identified, facilitating subsequent access and data transmission operations; by downloading sub-files from known identified data nodes, the accurate extraction of sub-files can be achieved, ensuring the consistency and integrity of downloaded data; and by combining sub-files with index information to generate the target file, the original file can be effectively reconstructed, ensuring the correctness and reliability of data restoration.

[0090] Figure 3 This invention provides a schematic diagram of a file upload device. This invention is applicable to situations where files are uploaded from a client to a server's data node. The device can execute a file upload method and can be implemented in hardware and / or software.

[0091] See Figure 3 The file upload device shown includes: a data acquisition module 301, a storage capacity calculation module 302, a target file splitting module 303, a target sub-file allocation module 304, and a target sub-file upload module 305, wherein...

[0092] The data acquisition module 301 is used to acquire network bandwidth rate, target file storage space and number of available data nodes;

[0093] The storage capacity calculation module 302 is used to calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes.

[0094] The target file splitting module 303 is used to split the target file according to the network bandwidth rate to obtain at least one target sub-file; the target file includes at least one target sub-file;

[0095] The target sub-file allocation module 304 is used to allocate each target sub-file to each available data node according to the storage capacity of each available data node, thereby obtaining the mapping relationship between each target sub-file and each available data node;

[0096] The target sub-file upload module 305 is used to upload each target sub-file to each available data node according to the mapping relationship.

[0097] The technical solution of this invention provides basic data support for subsequent steps by acquiring network bandwidth rate, target file size, and number of available data nodes. By calculating the storage capacity of each data node based on the storage space of the target file and the number of nodes, a balanced distribution of the target sub-file among the data nodes can be achieved, avoiding overload of a single node. By splitting the file according to network bandwidth, the size of the target sub-file can be matched with the actual network transmission capacity, thereby improving transmission efficiency and reducing the risk of network congestion during transmission. By allocating the target sub-file based on the storage capacity of the data nodes, distributed storage that better matches the load capacity of each node can be achieved. By uploading the target sub-file according to the mapping relationship, a one-to-one correspondence between the target sub-file and the available data nodes can be achieved, ensuring the correct distribution of the file. This invention solves the technical problem that random storage causes excessive storage pressure on a single data node, making it difficult to accurately track the storage location of each sub-file. This invention can improve the efficiency of file transmission.

[0098] In some embodiments, in allocating each target sub-file to each available data node according to the storage capacity in each available data node, and obtaining the mapping relationship between each target sub-file and each available data node, the target sub-file allocation module 304 is specifically used for:

[0099] The number of target sub-files that can be allocated in each available data node is determined based on the storage capacity in each available data node;

[0100] Based on the number of allocable target sub-files in each available data node, each target sub-file is assigned to each available data node, thus obtaining the mapping relationship between each target sub-file and each available data node.

[0101] In some embodiments, the data acquisition module 301 is specifically used to: Regarding the method by which the number of available data nodes can be obtained, the data acquisition module 301 is specifically configured to:

[0102] Obtain the connection status of each data node;

[0103] When the connection status of a data node is successful, the data node is marked as an available data node;

[0104] The number of available data nodes is obtained by statistically analyzing each available data node.

[0105] In some embodiments, the connection state includes at least one of the following: connection successful state, connecting state, and connection failed state.

[0106] In some embodiments, in uploading each target sub-file to each available data node according to the mapping relationship, the target sub-file upload module 305 is specifically used for:

[0107] Based on the mapping relationship, obtain the node identifier of each available data node;

[0108] Based on the node identifier, each target sub-file is uploaded to each available data node.

[0109] In some embodiments, after splitting the target file according to the network bandwidth rate to obtain at least one target sub-file, the target file splitting module is further specifically used for:

[0110] Based on the mapping relationship, generate index information for the target file; the index information records the concatenation order of each target sub-file in the target file.

[0111] In some embodiments, the target sub-file upload module 305 is further specifically used for:

[0112] Get download request;

[0113] Based on the download request, obtain the mapping relationship between each target sub-file and each available data node;

[0114] Based on the mapping relationship, obtain the identifier of each available data node;

[0115] Based on the identifier, download each target sub-file from each available data node;

[0116] Based on the target subfiles and index information, generate and transmit the target file.

[0117] The file upload device provided in the embodiments of the present invention can execute the file upload method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the file upload method.

[0118] Figure 4 This is a schematic diagram of a file upload device provided in an embodiment of the present invention.

[0119] like Figure 4 As shown, the file upload device 400 includes at least one processor 401 and a memory, such as a read-only memory (ROM) 402 and a random access memory (RAM) 403, communicatively connected to the at least one processor 401. The memory stores computer programs executable by the at least one processor. The processor 401 can perform various appropriate actions and processes based on the computer program stored in the ROM 402 or loaded into the RAM 403 from the storage unit 408. The RAM 403 can also store various programs and data required for the operation of the file upload device 400. The processor 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 408 is also connected to the bus 404.

[0120] Multiple components in the file upload device 400 are connected to the I / O interface 405, including: an input unit 406, such as a keyboard, mouse, etc.; an output unit 407, such as various types of displays, speakers, etc.; a storage unit 408, such as a disk, optical disk, etc.; and a communication unit 409, such as a network card, modem, wireless transceiver, etc. The communication unit 409 allows the file upload device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] Processor 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 401 performs the various methods and processes described above, such as file upload methods.

[0122] In some embodiments, the file upload method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the file upload device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by processor 401, one or more steps of the file upload method described above may be performed. Alternatively, in other embodiments, processor 401 may be configured to perform the file upload method by any other suitable means (e.g., by means of firmware).

[0123] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0124] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0125] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0126] To provide user interaction, the systems and techniques described herein can be implemented on the operational detection device. This file upload device includes: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the file upload device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0128] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0129] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A file upload method, characterized in that, The method includes: Obtain the network bandwidth rate, the storage space of the target file, and the number of available data nodes; Calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes; The target file is split according to the network bandwidth rate to obtain at least one target sub-file; the target file includes at least one target sub-file; the splitting is based on bandwidth adaptive rules. Based on the storage capacity of each available data node, each target sub-file is allocated to each available data node, thereby obtaining a mapping relationship between each target sub-file and each available data node; the mapping relationship records the storage target location of each target sub-file; According to the mapping relationship, each target sub-file is uploaded to each available data node; After splitting the target file according to the network bandwidth rate to obtain at least one target sub-file, the method further includes: Based on the mapping relationship, the index information of the target file is generated; the index information records the concatenation order of each target sub-file in the target file; Get download request; Based on the download request, obtain the mapping relationship between each target sub-file and each available data node; Based on the mapping relationship, obtain the identifier of each available data node; Based on the identifier, download each of the target sub-files in each of the available data nodes; The target file is generated and transmitted based on the target sub-file and the index information.

2. The method according to claim 1, characterized in that, The step of allocating each target sub-file to each available data node based on the storage capacity of each available data node, thereby obtaining the mapping relationship between each target sub-file and each available data node, includes: Based on the storage capacity of each available data node, determine the number of target sub-files that can be allocated in each available data node; Based on the number of target sub-files that can be allocated in each of the available data nodes, each target sub-file is allocated to each of the available data nodes, thereby obtaining the mapping relationship between each target sub-file and each of the available data nodes.

3. The method according to claim 1, characterized in that, The number of available data nodes is obtained in the following way: Obtain the connection status of each data node; When the connection status of the data node is successful, the data node is marked as an available data node; The number of available data nodes is obtained by statistically analyzing each of the aforementioned available data nodes.

4. The method according to claim 3, characterized in that, The connection state includes at least one of the following: connection successful, connection in progress, and connection failed.

5. The method according to claim 1, characterized in that, The step of uploading each target sub-file to each available data node according to the mapping relationship includes: Based on the mapping relationship, obtain the node identifier of each available data node; Based on the node identifier, each target sub-file is uploaded to each available data node.

6. A file upload device, characterized in that, include: The data acquisition module is used to obtain network bandwidth rate, target file storage space and number of available data nodes; The storage capacity calculation module is used to calculate the storage capacity of each available data node based on the storage space of the target file and the number of available data nodes; The target file splitting module is used to split the target file according to the network bandwidth rate to obtain at least one target sub-file; The target file includes at least one target sub-file; the splitting is based on bandwidth adaptive rules. The target sub-file allocation module is used to allocate each target sub-file to each available data node according to the storage capacity of each available data node, thereby obtaining the mapping relationship between each target sub-file and each available data node; The mapping relationship records the storage target location of each target sub-file; The target sub-file upload module is used to upload each of the target sub-files to each of the available data nodes according to the mapping relationship; After splitting the target file according to the network bandwidth rate to obtain at least one target sub-file, the device is further configured to generate index information of the target file according to the mapping relationship; the index information records the splicing order of each target sub-file in the target file; obtain a download request; obtain the mapping relationship between each target sub-file and each available data node according to the download request; obtain the identifier of each available data node according to the mapping relationship; According to the identifier, each target sub-file is downloaded in each of the available data nodes; according to the target sub-file and the index information, the target file is generated and transmitted.

7. A file upload device, characterized in that, The file upload device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the file upload method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the file upload method according to any one of claims 1-5.

Images