File transmission method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of data transmission, and discloses a file transmission method and device, electronic equipment and a storage medium, and the method comprises the steps: receiving an uploading request for a target file, and segmenting the target file into a preset number of sub-files according to the uploading request; selecting a target service node from the server, and allocating a corresponding target service node to each sub-file; creating uploading channels, and transmitting the corresponding sub-file to the corresponding target service node for storage by using each uploading channel; and generating a metadata file of the target file according to the first attribute information of the target file and the second attribute information of each sub-file, and storing the metadata file in a server for use when the target file is downloaded. According to the file transmission method disclosed by the invention, the file can be directly transmitted by utilizing the webpage of the browser without depending on the client application of the network disk, the file transmission efficiency can be improved, and the integrity and security of the data are also improved.

Description

Technical Field

[0001] The present invention relates to the field of data transmission, and in particular to a file transmission method, device, electronic equipment and storage medium. Background Art

[0002] With the rapid development of Internet technology, users' needs are becoming increasingly diversified and complex, and they are no longer satisfied with simple online information browsing. Users now prefer to manage and share their files through cloud storage services, which can not only transfer files to cloud storage, but also download them from the cloud to local devices.

[0003] At present, in the process of file transfer, most transmission technologies still use a single-channel upload method for a single file. Especially for large files, this upload method will lead to slow speed and long upload time. The process of downloading files also faces similar technical limitations. In addition, file transfer usually relies on the client / server (network disk) architecture to implement, that is, users need to install a specific network disk client application on their local device before calling the network disk API interface to complete file upload and download. This method may cause users to install multiple client applications when using network disk services from different providers, thereby occupying a large amount of storage space on the local device.

[0004] Therefore, how to improve the transmission efficiency of file transfer without relying on the client application of the network disk is a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of the above, it is necessary to provide a file transfer method, the purpose of which is to directly transfer files using the browser web page without relying on the client application of the network disk, and to improve the file transfer efficiency.

[0006] The file transmission method provided by the present invention comprises:

[0007] Receiving an upload request for a target file submitted by a user on a browser, and segmenting the target file into a preset number of sub-files according to the upload request;

[0008] Selecting a preset number of target service nodes from the server, and assigning a corresponding target service node to each subfile according to an identifier of each subfile, wherein the subfiles correspond to the target service nodes one by one, and the storage capacity of each target service node is greater than or equal to the file size of the corresponding subfile;

[0009] Based on the preset file upload method of the HTTP protocol, an upload channel is created between each sub-file and its corresponding target service node, and each upload channel is used to transfer the corresponding sub-file to the corresponding target service node for storage;

[0010] A metadata file of the target file is generated according to the first attribute information of the target file and the second attribute information of each sub-file, and the metadata file is stored in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

[0011] Optionally, the segmenting the target file into a preset number of sub-files according to the upload request includes:

[0012] Obtaining the file size of the target file, querying a preset fragmentation level table according to the file size of the target file, and obtaining the number of fragments into which the target file is divided;

[0013] The target file is divided according to the number of slices to obtain a preset number of sub-files.

[0014] Optionally, selecting a preset number of target service nodes from the server includes:

[0015] Acquire the coordinate information of each service node from the server, and acquire the coordinate information of the local device used by the user;

[0016] Calculating the location distance between the coordinate information of the local device and the coordinate information of each service node;

[0017] The service node whose location distance is less than the threshold is selected as the target service node.

[0018] Optionally, the preset file uploading method based on the HTTP protocol creates an upload channel between each sub-file and its corresponding target service node, including:

[0019] Based on the POST method of the HTTP protocol, configure a separate POST request for each sub-file;

[0020] The POST request is sent to the corresponding target service node for connection to obtain an upload channel.

[0021] The file transmission method provided by the present invention comprises:

[0022] Receive a download request for a target file submitted by a user on a browser, obtain a metadata file of the target file according to the download request, and obtain a file size, a number of sub-files, an identifier of each sub-file, and a storage path of each sub-file of the target file from the metadata file;

[0023] According to the file size of the target file, a pre-allocated file having a capacity greater than the file size of the target file is constructed on the local device of the user, the pre-allocated file is divided into partitions having the same number as the sub-files according to the number of sub-files, and a corresponding partition is allocated to each sub-file according to an identifier of each sub-file, wherein the sub-files correspond to the partitions one by one, and the storage capacity of each partition is greater than or equal to the file size of the corresponding sub-file;

[0024] Determine the target service node corresponding to each sub-file according to the storage path of each sub-file, create a download channel between each partition and its corresponding target service node based on the preset download method of the HTTP protocol, and use each download channel to transfer the corresponding sub-file to the corresponding partition for storage;

[0025] A first Hash value of the file size of the target file is calculated, and a second Hash value of all sub-files in the pre-allocated file is calculated. When the first Hash value is consistent with the second Hash value, all sub-files are merged to obtain the target file.

[0026] Optionally, constructing a pre-allocated file having a capacity greater than a file size of the target file on the local device of the user includes:

[0027] Creating a shell script, receiving the name and file size of the target file as parameters of the shell script;

[0028] Based on the preset command of the shell script, a pre-allocated file is created at a preset location of the user's local device, and the storage capacity of the pre-allocated file is automatically set to be larger than the file size of the target file.

[0029] Optionally, the download method preset based on the HTTP protocol creates a download channel between each partition and its corresponding target service node, including:

[0030] Based on the GET method of the HTTP protocol, a separate GET request is configured for each partition;

[0031] The GET request is sent to the corresponding target service node for connection to obtain a download channel.

[0032] In order to solve the above problem, the present invention further provides a file transmission device, the device comprising:

[0033] A receiving module, used for receiving an upload request for a target file submitted by a user on a browser, and segmenting the target file into a preset number of sub-files according to the upload request;

[0034] An allocation module, used to select a preset number of target service nodes from the server, and allocate a corresponding target service node to each sub-file according to an identifier of each sub-file;

[0035] The transmission module is used to create an upload channel between each sub-file and its corresponding target service node based on the preset file upload method of the HTTP protocol, and use each upload channel to transmit the corresponding sub-file to the corresponding target service node for storage;

[0036] A storage module is used to generate a metadata file of the target file based on the first attribute information of the target file and the second attribute information of each sub-file, and store the metadata file in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

[0037] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0038] at least one processor; and,

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a file transmission program executable by the at least one processor, and the file transmission program is executed by the at least one processor so that the at least one processor can execute the above-mentioned file transmission method.

[0041] In order to solve the above problem, the present invention further provides a computer-readable storage medium, on which a file transfer program is stored. The file transfer program can be executed by one or more processors to implement the above file transfer method.

[0042] Compared with the prior art, the file transmission method provided by the present invention divides a large target file into multiple small sub-files for uploading, and can upload these sub-files in parallel, thereby making full use of bandwidth resources and speeding up the uploading speed. Storing different parts of the sub-files on different target service nodes helps to balance the load of the server, and can reduce delays by selecting nodes that are geographically closer to users.

[0043] The file transfer method of the present invention can directly transfer files using the web page of the browser without relying on the client application of the network disk, and can improve the file transmission efficiency and enhance the integrity and security of the data. At the same time, it also simplifies the user's operation process and releases the memory space of the user's local device. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a process flow of a file transmission method provided by an embodiment of the present invention;

[0045] Figure 2 A flowchart of another embodiment of the file transmission method provided by one embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a module of a file transmission device provided by an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of the structure of an electronic device for implementing a file transmission method provided by an embodiment of the present invention;

[0048] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0051] Reference Figure 1 FIG. 1 is a flow chart of a file transmission method provided by an embodiment of the present invention. The method is executed by an electronic device.

[0052] In this embodiment, file transmission includes:

[0053] S1. Receive an upload request for a target file submitted by a user on a browser, and segment the target file into a preset number of sub-files according to the upload request.

[0054] In this embodiment, the user logs in to the web page interface of the server through the browser. A file selector is provided in the web page interface. The user selects the target file from the local device through the file selector. When the user clicks the "Upload" button, the browser sends an HTTP request containing basic information of the target file (such as the name of the target file, the total size of the target file, etc.) to the server.

[0055] The browser side refers to the user using a browser to access the web page interface provided by the server and upload files through this interface. The browser can be Google Chrome, Mozilla Firefox, Microsoft Edge or Safari.

[0056] The browser has built-in support for the HTTP protocol, and HTTP requests are initiated through simple JavaScript code, without the need for users to install additional software or plug-ins, nor the need to install the server's client application on the user's local device. That is, users only need to click the upload or download button on the server's web interface to implement file transfer and download functions through the web interface. In addition, through the HTTP protocol, multiple requests can be processed simultaneously, and multiple file segments can be uploaded or downloaded in parallel, thereby increasing the transmission speed.

[0057] The server refers to a backend system that provides target file transmission and download services. The server can be a physical server, a cluster of servers, or a service based on a cloud computing platform, and the present invention does not limit this.

[0058] After receiving the upload request, the server parses the basic file information in the upload request, obtains the file size of the target file, queries the preset fragmentation level table according to the file size of the target file, obtains the fragmentation size or the number of fragments of the split target file, and returns the obtained fragmentation size or the number of fragments of the split target file as a response to the browser. After analyzing the response, the browser obtains the fragmentation size or the number of fragments of the split target file, splits the target file according to the fragmentation size or the number of fragments, and obtains a preset number of sub-files.

[0059] For example, the default sharding level table is as follows:

[0060] File size range Number of Shards <100MB No sharding 100MB-1GB 10 pieces 1GB-10GB 20 tablets >10GB 100 pieces

[0061] If the total size of the target file is 2.5 GB, then according to the above sharding level table, the number of shards is 20, and the target file is sharded according to the number of shards, for example, a 2.5 GB file is divided into 20 125 MB sub-files.

[0062] In other embodiments, after receiving the upload request, the server obtains the file size of the target file, queries a preset sharding level table based on the file size of the target file, obtains the size of the shards, and divides the target file into pieces according to the size of the shards until the target file is divided and a preset number of sub-files are obtained.

[0063] For example, the default sharding level table is as follows:

[0064] File size range The size of each shard <100MB No sharding 100MB-1GB 10MB 1GB-10GB 50MB >10GB 100MB

[0065] If the total size of the target file is 2.5GB, then according to the above sharding level table, the size of each shard should be 50MB, and the number of shards is calculated as follows: number of shards = (total size of target file / size of each shard). Slice the target file, for example, split a 2.5GB file into 50 50MB sub-files.

[0066] In one embodiment, the step of segmenting the target file into a preset number of sub-files according to the upload request includes:

[0067] Obtaining the file size of the target file, querying a preset fragmentation level table according to the file size of the target file, and obtaining the number of fragments into which the target file is divided;

[0068] The target file is divided according to the number of slices to obtain a preset number of sub-files.

[0069] When the server receives an upload request from a user to upload a target file, it first parses the upload request and extracts the basic information of the target file, including the file size, and queries a pre-set sharding level table based on the file size. This sharding level table defines the recommended number of shards corresponding to different file size ranges.

[0070] The number of shards is returned as a response to the browser, and the number of sub-files (number of shards) into which the target file should be divided is determined according to the shard level table. The target file is divided into a corresponding number of sub-files according to the determined number of shards. Dividing the target file into a corresponding number of sub-files can improve the upload efficiency, especially when processing large files. This method is particularly suitable for multi-threaded or parallel upload technology, because multiple smaller sub-files can be uploaded at the same time, thereby speeding up the upload speed of the entire file.

[0071] In one embodiment, the step of segmenting the target file into a preset number of sub-files according to the upload request includes:

[0072] Obtaining the file size of the target file, querying a preset sharding level table according to the file size of the target file, and obtaining the size of the shards;

[0073] The target file is segmented piece by piece according to the size of the segments until the target file is segmented and a preset number of sub-files are obtained.

[0074] The present invention provides another sharding method: especially when the file size exceeds a certain threshold, the key point here is to determine the size of each shard based on the size of the file, rather than the number of shards.

[0075] A pre-set fragmentation level table is queried according to the size of the target file. According to the fragmentation level table, the server determines the size of each fragment into which the file should be divided, fragments the target file according to the size of the fragment, and fragments the target file piece by piece according to the determined size of each fragment, until the entire target file is completely divided into several sub-files.

[0076] S2. Select a preset number of target service nodes from the server, and assign a corresponding target service node to each subfile according to an identifier of each subfile, wherein the subfiles correspond to the target service nodes one-to-one, and the storage capacity of each target service node is greater than or equal to the file size of the corresponding subfile.

[0077] In this embodiment, the target file is divided into multiple smaller sub-files, each sub-file has a unique identifier (ID), a certain number of target service nodes are selected from the server, and it is ensured that the target service nodes have enough storage space to save the sub-files, and each sub-file is allocated to a designated target service node for storage according to its identifier. This ensures that the sub-files of the target file are dispersedly stored on multiple different service nodes, each sub-file is stored on a specific node, and can be located and accessed through the identifier of the sub-file.

[0078] In one embodiment, selecting a preset number of target service nodes from the server includes:

[0079] Acquire the coordinate information of each service node from the server, and acquire the coordinate information of the local device used by the user;

[0080] Calculating the location distance between the coordinate information of the local device and the coordinate information of each service node;

[0081] The service node whose location distance is less than the threshold is selected as the target service node.

[0082] The server has a list of service nodes. The service node list records the coordinate information of each service node with its geographical location. These coordinate information can be used to calculate the relative position between the service node and the user's device. The user's local device (such as a computer, mobile phone, etc.) also has geographical location information, which can be obtained through GPS, IP address, etc. The coordinate information of the local device is used to calculate the distance from the service node. The distance between the coordinates of the user device and the coordinates of each service node can be calculated by a certain geographical distance calculation formula, such as the spherical distance formula or the Euclidean distance in the plane rectangular coordinate system. Select those service nodes whose location distance is less than a certain threshold from all service nodes. This threshold can be set according to actual needs and network topology. In order to improve the efficiency of file transfer and reduce network latency, the server will select those service nodes that are geographically close to the user to store the file fragments, which can reduce the distance of data transmission, thereby reducing network latency and increasing upload speed.

[0083] In one embodiment, selecting a preset number of target service nodes from the server includes:

[0084] Using a preset load balancer to monitor the load status of each service node in the server;

[0085] A service node whose load status is less than a threshold and has a file size sufficient to store each sub-file is selected as a target service node.

[0086] The server is deployed with a load balancer component, which is responsible for monitoring the load status of all service nodes. The load status can refer to indicators such as CPU usage, memory usage, and disk space usage on the service node. By continuously monitoring these indicators, the load balancer can understand in real time which service nodes are in a low load state and which are in a high load state. When a service node needs to be selected to store a sub-file, the load balancer will filter out those service nodes whose load status is lower than a certain threshold from the service node list. This threshold can be a fixed value or dynamically adjusted according to the current overall system load. For the low-load service nodes that have been selected, it is also necessary to check whether they have enough storage space to store the sub-files to be uploaded. Only those service nodes that meet both low load and sufficient storage space will be selected as target service nodes.

[0087] In one embodiment, allocating a corresponding target service node to each sub-file according to the identifier of each sub-file includes:

[0088] According to the identifier of the target service node, all the target service nodes are sorted to obtain a polling selection table;

[0089] According to the order of the polling selection table, each sub-file is allocated to each target service node in the polling selection table one by one until all sub-files are allocated.

[0090] The server will collect the identifiers of all available target service nodes, and these service node identifiers will be sorted in a certain order (for example, alphabetical order, ID order, etc.). The sorted service node list constitutes a polling selection table. According to the sorting of the polling selection table, each subfile is assigned to each target service node in the polling selection table one by one until all subfiles are assigned. The server will traverse the polling selection table, starting from the first service node, assign the first subfile to it, and then continue to the next service node to assign the second subfile, and so on. If the polling reaches the end of the list, it will return to the first service node in the list to continue the assignment until all subfiles are assigned. In order to ensure that each subfile can be effectively assigned to different service nodes, the server will create a polling selection table containing all target service nodes, and assign subfiles to service nodes one by one according to the order of this table. This method helps to achieve load balancing and ensure that no single service node carries too many subfiles, thereby improving the stability and reliability of the system.

[0091] S3. Based on the preset file upload method of the HTTP protocol, an upload channel is created between each sub-file and its corresponding target service node, and each upload channel is used to transfer the corresponding sub-file to the corresponding target service node for storage.

[0092] In this embodiment, the browser has built-in support for the HTTP protocol, and HTTP requests are initiated through simple JavaScript code, without the need for users to install additional software or plug-ins, nor the need to install the server's client application on the user's local device, that is, the user only needs to click the upload or download button on the server's web page interface to implement file transfer and download functions through the web page interface. In addition, through the HTTP protocol, multiple requests can be processed simultaneously, and multiple file segments can be uploaded or downloaded in parallel, thereby increasing the transmission speed.

[0093] A dedicated upload channel is created for each sub-file based on the POST method of the HTTP protocol. This channel connects the user's browser where the sub-file is located and the target service node (the server used to store the sub-file). Each upload channel is used to transfer each sub-file from the browser to the corresponding target service node. When the sub-file reaches the target service node through the upload channel, the service node will store the sub-file in its local storage.

[0094] Uploading multiple sub-files in parallel through different upload channels can significantly improve the overall speed of target file transmission. By distributing sub-files on different target service nodes, the server load can be effectively dispersed to avoid overloading a single node.

[0095] In one embodiment, the preset file uploading method based on the HTTP protocol creates an upload channel between each sub-file and its corresponding target service node, including:

[0096] Based on the POST method of the HTTP protocol, configure a separate POST request for each sub-file;

[0097] The POST request is sent to the corresponding target service node for connection to obtain an upload channel.

[0098] Based on the preset file upload method of the HTTP protocol, the process of creating an upload channel between each sub-file and its corresponding target service node is as follows:

[0099] For each sub-file, the server creates an HTTP POST request. A POST request is an HTTP request method used to send data to the server, usually used to submit a form or upload a file. The POST request for each sub-file will be sent to the target service node assigned to it. When sending the request, each sub-file establishes a connection with the service node to form an independent upload channel. Through this upload channel, the data of the sub-file will be transferred to the service node for storage.

[0100] Effectively utilize the characteristics of the HTTP protocol to achieve efficient file upload, while also ensuring the stability and reliability of the system. This method is particularly suitable for situations where a large number of fragments need to be processed, because it allows multiple sub-files to be uploaded in parallel, thereby speeding up the overall upload speed.

[0101] In one embodiment, the method further comprises:

[0102] Create a separate thread for each sub-file according to the identifier of each sub-file and the identifier of its corresponding target service node;

[0103] The startup thread is connected to the corresponding target service node to improve the stability when the sub-file is transmitted to the corresponding target service node.

[0104] For each sub-file, the server will create an independent thread, each thread is associated with the corresponding sub-file and its corresponding target service node. The identifier here refers to the unique identifier of the sub-file and the target service node, which can be the file name, file ID or service node ID.

[0105] Each thread is responsible for establishing a connection with the corresponding target service node. This connection can ensure the stability and transmission efficiency of the upload channel. Through this channel, the data of the sub-file can be transmitted to the target service node. Taking advantage of the multi-threading technology, multiple sub-files can be uploaded at the same time, thereby improving the overall efficiency of file transfer. By creating an independent thread for each sub-file, concurrent uploads can be achieved, reducing the total upload time and improving the system throughput. This method is particularly suitable for processing large files or multi-shard files.

[0106] S4. Generate a metadata file of the target file according to the first attribute information of the target file and the second attribute information of each sub-file, and store the metadata file in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

[0107] In this embodiment, some basic information about the target file is obtained, including the file name, file size, creation date, number of sub-files after sharding, etc. These basic information are used as the first attribute information of the target file. Some relevant information about each sub-file is obtained, including the file size identifier and storage path. These basic information are used as the second attribute information of each sub-file. The first attribute information of the target file and the second attribute information of each sub-file are obtained to verify the integrity of the data and to correctly reorganize these sub-files back to the original target file in the future.

[0108] A metadata file of the target file is generated based on the first attribute information of the target file and the second attribute information of each subfile, and the generated metadata file is stored on the server to facilitate access to this information when needed, especially when a user requests to download the target file.

[0109] Reference Figure 2 As shown, the file transmission method also includes:

[0110] S5. Receive a download request for a target file submitted by a user on a browser, obtain a metadata file of the target file according to the download request, and obtain a file size, a number of sub-files, an identifier of each sub-file, and a storage path of each sub-file of the target file from the metadata file.

[0111] In this embodiment, the user submits a download request on the browser side to download the target file. After receiving the download request, the server first searches for the metadata file related to the target file. The metadata file is created during the target file transmission process, and contains important information about the target file and its sub-files.

[0112] Parse the metadata file and extract the necessary information from the metadata file, mainly including: the file size of the target file. The number of sub-files: the number of sub-files into which the target file is divided. The identifier of each sub-file: information used to uniquely identify each sub-file, which can be a file name or a unique ID. The storage path of each sub-file: the specific location of the sub-file on the server, which is used to read these sub-files later. After obtaining the above necessary information, the download process can be prepared based on this necessary information, which involves retrieving sub-files from different storage locations and combining them in the correct order to form the original target file.

[0113] In one embodiment, the step of obtaining the metadata file of the target file according to the download request includes:

[0114] Determining the address of the server according to the URL included in the download request;

[0115] Requesting the metadata file of the target file from the address of the server in a GET manner based on the HTTP protocol;

[0116] Receive the HTTP response returned by the server to the GET method, parse the HTTP response, and obtain the metadata file of the target file.

[0117] The metadata file of the target file is obtained according to the download request of the user. The specific process is as follows: 1. Determine the address of the server according to the URL contained in the download request: The user sends a download request through the browser. The request contains the URL pointing to the metadata file of the target file. The address of the server, that is, the location of the server providing the metadata file, can be parsed from the URL. 2. Request the metadata file of the target file from the address of the server based on the GET method of the HTTP protocol: The user's browser will use the HTTP GET method to send a request to the server. The target of the request is the metadata file. This file contains all the necessary information about the target file, such as the file size, the number of fragments, the identifier of each sub-file and the storage path, etc. 3. Receive the HTTP response returned by the server to the GET method, parse the HTTP response, and obtain the metadata file of the target file: After receiving the GET request, the server will process the request and return an HTTP response. This response contains the content of the requested metadata file. The user's browser will receive the HTTP response, parse the content in the response body, and extract the metadata file from it.

[0118] S6. According to the file size of the target file, construct a pre-allocated file with a capacity larger than the file size of the target file on the local device of the user, divide the pre-allocated file into partitions with the same number as the sub-files according to the number of sub-files, and allocate a corresponding partition to each sub-file according to an identifier of each sub-file, wherein the sub-files correspond to the partitions one by one, and the storage capacity of each partition is larger than or equal to the file size of the corresponding sub-file;

[0119] In this embodiment, in the process of the user downloading the target file, how to prepare to receive the target file on the user's local device. It can be implemented by dividing it into several key steps: 1. Create a pre-allocated file: According to the file size of the target file, create a pre-allocated file on the user's local device. The capacity of this pre-allocated file needs to be greater than or equal to the file size of the target file. The role of this pre-allocated file is to reserve enough space to store the file to be downloaded to prevent the download from failing due to insufficient disk space during the download process. 2. Divide the pre-allocated file: According to the number of sub-files into which the target file is divided, divide the pre-allocated file into the same number of partitions. Each partition will be used to store one sub-file. For example, if the target file is divided into 100 sub-files, the pre-allocated file will also be divided into 100 partitions. 3. Allocate partitions: According to the identifier of each sub-file, each sub-file is allocated to a specific partition in the pre-allocated file. This means that each sub-file has a corresponding partition for storage.

[0120] By pre-allocating file space, the time for dynamically allocating space during the download process can be reduced, thereby speeding up the download. By dividing the pre-allocated file into partitions equal to the number of sub-files, it can be ensured that each sub-file is accurately stored in the correct location in the pre-allocated file. Each sub-file has a corresponding partition, which makes it easier to reassemble the sub-files into the original target file after the download is completed.

[0121] In one embodiment, constructing a pre-allocated file having a capacity greater than the file size of the target file on the local device of the user includes:

[0122] Creating a shell script, receiving the name and file size of the target file as parameters of the shell script;

[0123] Based on the preset command of the shell script, a pre-allocated file is created at a preset location of the user's local device, and the storage capacity of the pre-allocated file is automatically set to be larger than the file size of the target file.

[0124] The process of creating a pre-allocated file on the user's local device is to reserve sufficient storage space for the target file to be downloaded. Specifically, it involves using a shell script to create the pre-allocated file, as follows: 1. Create a shell script that receives the name and file size of the target file as parameters of the shell script: It is necessary to create a shell script that can receive two parameters: the name and file size of the target file. The user can run this script through a command line tool (such as a terminal or command prompt) and pass these two parameters. 2. Based on the preset command of the shell script, create a pre-allocated file at a preset location on the user's local device, and automatically set the storage capacity of the pre-allocated file to be larger than the file size of the target file: The dd command is used in the shell script to create the pre-allocated file. dd is a Linux / Unix command that is commonly used to copy files and convert and copy files. Use the dd command to create an empty file and pre-allocate a certain amount of storage space to ensure that the file occupies enough disk space when it is created. The size of the pre-allocated file will be set to be slightly larger than the file size of the target file to reserve some additional space. In this way, you can reserve enough disk space for file download in advance to avoid the problem of insufficient disk space during the download process, thereby improving the efficiency and reliability of the download process.

[0125] S7. Determine the target service node corresponding to each sub-file according to the storage path of each sub-file, create a download channel between each partition and its corresponding target service node based on the preset download method of the HTTP protocol, and use each download channel to transfer the corresponding sub-file to the corresponding partition for storage.

[0126] In this embodiment, the target service node for storing each sub-file is determined based on the storage path of each sub-file in the metadata file. This means that each sub-file has a specific storage location, that is, it is located on a certain target service node. A download channel is created between each sub-file and its corresponding target service node based on the GET method of the HTTP protocol. This is equivalent to creating a dedicated connection for each sub-file for downloading the sub-file from the target service node. Each download channel is used to download each sub-file from its corresponding target service node to a pre-allocated partition in a pre-allocated file on the local device in a parallel download manner. By downloading sub-files in parallel, the download speed can be significantly improved. By downloading each sub-file to a specific partition in the pre-allocated file, the integrity and accuracy of the final file can be ensured. Each sub-file is downloaded to a specific location in the pre-allocated file, which makes it easier to reorganize the target file.

[0127] In one embodiment, determining the target service node corresponding to each sub-file according to the storage path of each sub-file includes:

[0128] Get the identifier of the storage path of each sub-file;

[0129] The server is queried according to the identifier of the storage path of each sub-file to obtain the target service node corresponding to each sub-file.

[0130] The details are as follows: 1. Obtain the identifier of the storage path of each sub-file: The metadata file contains the storage path information of each sub-file. The identifier of the storage path can help determine on which service node the sub-file is stored. For example, the storage path may contain the ID or name of the service node, or other information that uniquely identifies the service node. 2. Query the server according to the identifier of the storage path of each sub-file to obtain the target service node corresponding to each sub-file: Query the storage system according to the identifier of the storage path to determine the actual storage location of each sub-file. The query result will return the information of the service node corresponding to each sub-file. This query process can be completed by querying the database, cache or other storage mechanisms. In this way, the load of the service node can be effectively managed to ensure that resources are reasonably allocated, while also improving the overall performance of the system and user satisfaction.

[0131] In one embodiment, the download method preset based on the HTTP protocol creates a download channel between each partition and its corresponding target service node, including:

[0132] Based on the GET method of the HTTP protocol, a separate GET request is configured for each partition;

[0133] The GET request is sent to the corresponding target service node for connection to obtain a download channel.

[0134] The details are as follows: 1. Based on the GET method of the HTTP protocol, each partition is treated as a separate GET request: For each partition in the pre-allocated file, an HTTP GET request is created, and each GET request is used to request the sub-file data stored on the corresponding service node. 2. The GET request is sent to the corresponding target service node for connection to obtain a download channel: The GET request for each partition will be sent to the target service node assigned to it. When sending the request, the client establishes a connection with the service node to form a download channel. Through this download channel, the sub-file data on the service node will be transmitted to the corresponding partition in the pre-allocated file of the client.

[0135] In one embodiment, the method further comprises:

[0136] Create a separate thread for each partition based on the identifier of each partition and the identifier of its corresponding target service node;

[0137] The startup thread connects to the corresponding target service node to improve the stability of downloading sub-files.

[0138] Create a separate thread for each partition based on the identifier of each partition and the identifier of its corresponding target service node: For each partition in the pre-allocated file, create a separate thread based on the identifier of the partition and the identifier of the corresponding target service node. Each thread is associated with a partition and a target service node. The identifier here can be a unique identifier of the partition and the target service node, such as a partition number, a service node ID, etc.

[0139] Each thread is responsible for establishing a connection with the corresponding target service node. This connection can ensure the stability and transmission efficiency of the download channel. The sub-file data on the service node can be transferred to the corresponding partition in the client's pre-allocated file. Once the connection is established, the sub-file data can be downloaded to the corresponding partition in the pre-allocated file through this download channel.

[0140] Taking advantage of multi-threading technology, it allows multiple sub-files to be downloaded simultaneously, thus improving the overall efficiency of file downloading. By creating independent threads for each partition, concurrent downloads can be achieved, reducing the total download time and improving the system throughput. This method is particularly suitable for processing large files or multi-shard files.

[0141] S8. Calculate a first Hash value of the file size of the target file, and calculate a second Hash value of all sub-files in the pre-allocated file. When the first Hash value is consistent with the second Hash value, merge all sub-files to obtain the target file.

[0142] In this embodiment, after all sub-files are downloaded, the first hash value of the target file is calculated. After the sub-files are downloaded, the second hash values ​​of all sub-files in the pre-allocated file are calculated. The second hash value is a summary of the contents of all sub-files in the pre-allocated file. The first hash value is compared with the second hash value. If the two hash values ​​are completely consistent, it can be confirmed that the downloaded sub-files completely match the original target file and no data damage or loss occurs. If the two hash values ​​are consistent, that is, the hash value of the downloaded sub-file is the same as the hash value of the original target file, then it can be considered that the download process is successful, and all sub-files can be merged into the original target file.

[0143] By comparing hash values, you can ensure that the downloaded sub-files are exactly the same as the original target file without data corruption or loss. Once the hash values ​​are confirmed to be consistent, you can safely merge the sub-files to get the original target file.

[0144] In one embodiment, calculating the first Hash value of the file size of the target file includes:

[0145] Read the content data of the target file and create an MD5 hash object;

[0146] The content data of the target file is calculated using the MD5 hash object to obtain a first hash value of the target file.

[0147] The content data of the file needs to be read. This usually involves opening the file and reading the contents of the file block by block. MD5 is a widely used hash function that is used to generate a fixed-length digest. An MD5 hash object is created to convert the content data of the file into a hash value, and this hash object will be used to process the read file content data. The content data of the target file is calculated using the MD5 hash object to obtain the first hash value of the target file: while reading the file content data, the data block is passed to the MD5 hash object for processing. The MD5 hash object will gradually update its internal state until all the file content data is processed. Finally, the MD5 hash object will generate a fixed-length hash value, which is the first hash value of the target file.

[0148] During the file integrity verification process, in order to ensure the integrity of the target file, the first hash value of the file size of the target file is calculated. The first hash value can be used as a digital fingerprint of the file content to verify whether the file has not been modified.

[0149] The second hash value of the sub-file is calculated using the MD5 hash method. The above-mentioned MD5 hash method of the first hash value of the file size of the target file can be referred to. The implementation methods of the two are the same and will not be repeated here.

[0150] like Figure 3 FIG. 1 is a schematic diagram of a module of a file transmission device provided by an embodiment of the present invention.

[0151] The file transmission device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the file transmission device 100 may include a receiving module 110, a distribution module 120, a transmission module 130 and a storage module 140. The module of the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in the memory of the electronic device.

[0152] In this embodiment, the functions of each module / unit are as follows:

[0153] The receiving module 110 is used to receive an upload request for a target file submitted by a user on a browser, and to segment the target file into a preset number of sub-files according to the upload request;

[0154] An allocation module 120 is used to select a preset number of target service nodes from the server, and allocate a corresponding target service node to each subfile according to an identifier of each subfile, wherein the subfiles correspond to the target service nodes one by one, and the storage capacity of each target service node is greater than or equal to the file size of the corresponding subfile;

[0155] The transmission module 130 is used to create an upload channel between each sub-file and its corresponding target service node based on the preset file upload method of the HTTP protocol, and use each upload channel to transmit the corresponding sub-file to the corresponding target service node for storage;

[0156] The storage module 140 is used to generate a metadata file of the target file based on the first attribute information of the target file and the second attribute information of each sub-file, and store the metadata file in the server for use when downloading the target file. The first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

[0157] like Figure 4 , which is a schematic diagram of the structure of an electronic device for implementing a file transmission method provided by an embodiment of the present invention.

[0158] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a file transfer program 10 that can be executed by the processor 12. Figure 4 Only the electronic device 1 having the components 11-13 and the file transfer program 10 is shown, and those skilled in the art will understand that Figure 4 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0159] The memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. equipped on the electronic device 1. In this embodiment, the readable storage medium of the memory 11 is generally used to store the operating system and various application software installed on the electronic device 1, such as storing the code of the file transfer program 10 in an embodiment of the present invention. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.

[0160] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the file transfer program 10.

[0161] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a terminal (not shown in the figure).

[0162] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0163] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0164] The file transfer program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0165] Receiving an upload request for a target file submitted by a user on a browser, and segmenting the target file into a preset number of sub-files according to the upload request;

[0166] Selecting a preset number of target service nodes from the server, and assigning a corresponding target service node to each subfile according to an identifier of each subfile, wherein the subfiles correspond to the target service nodes one by one, and the storage capacity of each target service node is greater than or equal to the file size of the corresponding subfile;

[0167] Based on the preset file upload method of the HTTP protocol, an upload channel is created between each sub-file and its corresponding target service node, and each upload channel is used to transfer the corresponding sub-file to the corresponding target service node for storage;

[0168] A metadata file of the target file is generated according to the first attribute information of the target file and the second attribute information of each sub-file, and the metadata file is stored in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

[0169] Specifically, the specific implementation method of the processor 12 for the above-mentioned file transfer program 10 can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0170] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0171] The computer-readable storage medium stores a file transfer program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-mentioned file transfer method embodiments, and will not be described in detail here.

[0172] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0173] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0175] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0176] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0177] It should be noted that if software tools or components other than those of our company appear in the embodiments of the present application, they are only used for illustrative purposes and do not represent actual use.

[0178] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0179] Finally, it should be noted that the serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0180] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A file transmission method, characterized in that: The uploading method includes: Receiving an upload request for a target file submitted by a user on a browser, and segmenting the target file into a preset number of sub-files according to the upload request; Selecting a preset number of target service nodes from the server, and assigning a corresponding target service node to each subfile according to an identifier of each subfile, wherein the subfiles correspond to the target service nodes one by one, and the storage capacity of each target service node is greater than or equal to the file size of the corresponding subfile; Based on the preset file upload method of the HTTP protocol, an upload channel is created between each sub-file and its corresponding target service node, and each upload channel is used to transfer the corresponding sub-file to the corresponding target service node for storage; A metadata file of the target file is generated according to the first attribute information of the target file and the second attribute information of each sub-file, and the metadata file is stored in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

2. The file transmission method according to claim 1, characterized in that: The step of segmenting the target file into a preset number of sub-files according to the upload request includes: Obtaining the file size of the target file, querying a preset fragmentation level table according to the file size of the target file, and obtaining the number of fragments into which the target file is divided; The target file is divided according to the number of slices to obtain a preset number of sub-files.

3. The file transmission method according to claim 1, characterized in that: The selecting a preset number of target service nodes from the server includes: Acquire the coordinate information of each service node from the server, and acquire the coordinate information of the local device used by the user; Calculating the location distance between the coordinate information of the local device and the coordinate information of each service node; The service node whose location distance is less than the threshold is selected as the target service node.

4. The file transmission method according to claim 1, characterized in that: The preset file uploading method based on the HTTP protocol creates an upload channel between each sub-file and its corresponding target service node, including: Based on the POST method of the HTTP protocol, configure a separate POST request for each sub-file; The POST request is sent to the corresponding target service node for connection to obtain an upload channel.

5. The file transmission method according to claim 1, characterized in that: The file transmission method further includes: Receive a download request for a target file submitted by a user on a browser, obtain a metadata file of the target file according to the download request, and obtain a file size, a number of sub-files, an identifier of each sub-file, and a storage path of each sub-file of the target file from the metadata file; According to the file size of the target file, a pre-allocated file having a capacity greater than the file size of the target file is constructed on the local device of the user, the pre-allocated file is divided into partitions having the same number as the sub-files according to the number of sub-files, and a corresponding partition is allocated to each sub-file according to an identifier of each sub-file, wherein the sub-files correspond to the partitions one by one, and the storage capacity of each partition is greater than or equal to the file size of the corresponding sub-file; Determine the target service node corresponding to each sub-file according to the storage path of each sub-file, create a download channel between each partition and its corresponding target service node based on the preset download method of the HTTP protocol, and use each download channel to transfer the corresponding sub-file to the corresponding partition for storage; A first Hash value of the file size of the target file is calculated, and a second Hash value of all sub-files in the pre-allocated file is calculated. When the first Hash value is consistent with the second Hash value, all sub-files are merged to obtain the target file.

6. The file transmission method according to claim 5, characterized in that: The step of constructing a pre-allocated file with a capacity larger than the file size of the target file on the local device of the user includes: Creating a shell script, receiving the name and file size of the target file as parameters of the shell script; Based on the preset command of the shell script, a pre-allocated file is created at a preset location of the user's local device, and the storage capacity of the pre-allocated file is automatically set to be larger than the file size of the target file.

7. The file transmission method according to claim 5, characterized in that: The download method preset based on the HTTP protocol creates a download channel between each partition and its corresponding target service node, including: Based on the GET method of the HTTP protocol, a separate GET request is configured for each partition; The GET request is sent to the corresponding target service node for connection to obtain a download channel.

8. A file transmission device, characterized in that: The device comprises: A receiving module, used for receiving an upload request for a target file submitted by a user on a browser, and segmenting the target file into a preset number of sub-files according to the upload request; An allocation module, used to select a preset number of target service nodes from the server, and allocate a corresponding target service node to each sub-file according to an identifier of each sub-file; The transmission module is used to create an upload channel between each sub-file and its corresponding target service node based on the preset file upload method of the HTTP protocol, and use each upload channel to transmit the corresponding sub-file to the corresponding target service node for storage; A storage module is used to generate a metadata file of the target file based on the first attribute information of the target file and the second attribute information of each sub-file, and store the metadata file in the server for use when downloading the target file, wherein the first attribute information includes the file size of the target file and the number of sub-files after segmentation, and the second attribute information includes the file size, identifier and storage path.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a file transfer program executable by the at least one processor, and the file transfer program is executed by the at least one processor so that the at least one processor can execute the file transfer method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a file transfer program, and the file transfer program can be executed by one or more processors to implement the file transfer method according to any one of claims 1 to 7.

Images