File transmission method and device

Abstract

The invention provides a file transmission method and device, and the method comprises the steps: dividing a to-be-uploaded file into a plurality of fragments to generate a first fragment set under the condition that the to-be-uploaded file meets a breakpoint resume condition, and generating an uploading secret key for the first fragment set; according to the uploading secret key and each fragment parameter in the first fragment set, screening a historical fragment set meeting a condition from historical uploading information of the file; searching a matched second fragment set according to the parameter information of each fragment in the first fragment set, and determining a difference fragment between the first fragment set and the second fragment set; and uploading data corresponding to the differential fragments by adopting multiple threads, and merging the second fragment set and the data corresponding to the differential fragments into a target file. According to the file transmission scheme provided by the invention, the file can be uploaded through a breakpoint resuming method, so that the historically uploaded fragments of the same file are utilized, the problem that the historically uploaded fragments occupy additional storage space can be avoided, and traffic resources consumed by file uploading can also be saved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a file transmission method and device. Background Art

[0002] With the rapid development of cloud storage technology, more flexible and convenient services are provided for user storage. When users upload large files based on traditional object storage protocols, the files are segmented into multiple segments, and a request is generated for each segment to upload to the server, thus avoiding request timeout and memory overflow problems.

[0003] However, in actual applications, when uploading pieces synchronously based on the object storage protocol, if the synchronization process stops during the transmission of the pieces into which the large file is divided, some pieces will be uploaded to the server, while some pieces will not be uploaded to the server, and the uploaded pieces cannot be displayed as files. When the user uploads the large file again, some of the previously uploaded pieces will not be used, wasting storage space and traffic resources consumed by file uploads. Summary of the invention

[0004] The present application provides a file transmission method and device, which can solve the problem in the prior art that the file fragments of historical uploads cannot be utilized, thereby wasting storage space and traffic resources consumed by file uploads.

[0005] According to a first aspect of the present application, a file transmission method is provided, wherein the method comprises:

[0006] When the file to be uploaded meets the breakpoint resumable upload condition, the file is divided into a plurality of fragments according to a first preset rule to generate a first fragment set and an upload key is generated for the first fragment set;

[0007] Filtering a historical shard set that meets the conditions from historical upload information of the file according to the upload key and the parameters of each shard in the first shard set;

[0008] According to the parameter information of each shard in the first shard set, searching for a matching second shard set from the historical shard set that meets the conditions;

[0009] Determine different shards in the first shard set and the second shard set;

[0010] The data corresponding to the differential shards are uploaded using multiple threads, and the second shard set and the data corresponding to the differential shards are merged into a target file.

[0011] In some embodiments, the method further comprises:

[0012] Determine the buffer area allocated for the user;

[0013] Querying whether there is a historical upload record of the file to be uploaded in the cache area;

[0014] If so, determine that the file to be uploaded meets the breakpoint resume condition;

[0015] If it does not exist, it is determined that the file to be uploaded does not meet the breakpoint resume condition.

[0016] In some implementations, the shard parameter information includes: shard size, shard index, service identifier corresponding to the shard, and hash value of the shard.

[0017] In some implementations, the step of searching for a matching second shard set from the historical shard set that meets the conditions according to the parameter information of each shard in the first shard set includes:

[0018] For each historical shard set that meets the condition, matching the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one;

[0019] Determining the number of shards in the historical shard set that match the first shard set;

[0020] A historical shard set having the largest number of shards matching the first shard set is determined as the second shard set.

[0021] In some implementations, for each historical shard set that meets the condition, the step of matching parameter information of shards in the historical shard set with parameter information of shards in the first shard set one by one includes:

[0022] For each of the historical shard sets that meets the condition, extracting shard parameter information of a shard from the shard set;

[0023] Comparing the shard subscript of the shard with the shard subscripts of the shards in the first shard set;

[0024] If the two are the same, the hash values ​​of the shard pair are compared;

[0025] If the hash values ​​of the shard pair are the same, then it is determined that the shard pair matches.

[0026] In some implementations, the step of determining the difference shards in the first shard set and the second shard set includes:

[0027] Determine matching shard pairs in the first shard set and the second shard set;

[0028] Deleting the shards in the second shard set that are not matched with the shards in the first shard set to obtain a third shard set;

[0029] Each shard in a non-intersection of the first shard set and the third shard set is determined as a differential shard.

[0030] According to a second aspect of the present application, a file transmission device is provided, wherein the device comprises:

[0031] A fragmentation module, configured to, when a file to be uploaded meets a breakpoint-resume upload condition, divide the file into a plurality of fragments according to a first preset rule to generate a first fragment set and generate an upload key for the first fragment set;

[0032] An information determination module, configured to filter a set of historical shards that meet conditions from historical upload information of the file according to the upload key and parameters of each shard in the first shard set;

[0033] A search module, configured to search for a matching second shard set from the historical shard set that meets the conditions according to the parameter information of each shard in the first shard set;

[0034] A difference determination module, configured to determine different shards between the first shard set and the second shard set;

[0035] The resuming module is used to upload the data corresponding to the differential shards by using multiple threads, and merge the data corresponding to the second shard set and the differential shards into a target file.

[0036] In some embodiments, the device further comprises:

[0037] A cache area determination module, used to determine the cache area allocated to the user;

[0038] A query module, used to query whether there is a historical upload record of the file to be uploaded in the cache area;

[0039] A query result determination module, used to determine if the file to be uploaded meets the breakpoint resume condition;

[0040] If it does not exist, it is determined that the file to be uploaded does not meet the breakpoint resume condition.

[0041] In some implementations, the shard parameter information includes: shard size, shard index, service identifier corresponding to the shard, and hash value of the shard.

[0042] In some implementations, the search module includes:

[0043] A first submodule is used to match the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one for each historical shard set that meets the condition;

[0044] A second submodule is used to determine the number of shards in the historical shard set that match the first shard set;

[0045] The third submodule is used to determine the historical shard set with the largest number of shards matching the first shard set as the second shard set.

[0046] In some embodiments, the first submodule includes:

[0047] The first unit is configured to extract, for each of the historical shard sets that meets a condition, shard parameter information of a shard from the shard set;

[0048] A second unit is used to compare the shard subscript of the shard with the shard subscripts of the shards in the first shard set;

[0049] The third unit is used for comparing the hash values ​​of the shard pairs if there are two identical shard pairs;

[0050] The fourth unit is used to determine that the shard pair matches if the hash values ​​of the shard pair are the same.

[0051] In some embodiments, the difference determination module includes:

[0052] A fourth submodule, configured to determine matching shard pairs in the first shard set and the second shard set;

[0053] A fifth submodule, configured to delete the shards in the second shard set that are not matched with the shards in the first shard set, to obtain a third shard set;

[0054] The sixth submodule is configured to determine each shard in a non-intersection of the first shard set and the third shard set as a different shard.

[0055] According to the third aspect of the present application, there is provided an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; the memory is used to store computer programs; and the processor is used to implement any one of the above-mentioned file transfer methods when executing the program stored in the memory.

[0056] According to a fourth aspect of the present application, a computer storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, any of the above-mentioned file transfer methods is implemented.

[0057] In summary, the file transmission method and device provided by the present application have at least the following beneficial effects:

[0058] The file transmission method of the present application includes: when the file to be uploaded meets the breakpoint resume condition, the file is divided into multiple fragments according to the first preset rule to generate a first fragment set and an upload key is generated for the first fragment set; according to the upload key and the parameters of each fragment in the first fragment set, the historical fragment set that meets the condition is screened from the historical upload information of the file; according to the parameter information of each fragment in the first fragment set, a matching second fragment set is searched from the historical fragment set that meets the condition; the difference fragments in the first fragment set and the second fragment set are determined; the data corresponding to the difference fragments are uploaded by multi-threading, and the data corresponding to the second fragment set and the difference fragments are merged into the target file. The file transmission method provided by the present application uploads files by the breakpoint resume method, so that the fragments of the same file that have been uploaded in the past are used, which can not only avoid the problem of the historical uploaded fragments occupying extra storage space, but also save the traffic resources consumed by file upload. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 is a flowchart showing the steps of a file transmission method according to an embodiment of the present application;

[0061] Figure 2 is a structural block diagram of a file transmission device according to an embodiment of the present application;

[0062] Figure 3 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the above and other features and advantages of the present application more clear, the present application is further described below in conjunction with the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0064] In the following description, many specific details are set forth to provide a thorough understanding of the present application. However, it is apparent to those skilled in the art that specific details need not be adopted to practice the present application. In other cases, well-known steps or operations are not described in detail to avoid blurring the present application.

[0065] The file transfer method provided in the embodiment of the present application can be applied to an electronic device, on which a computer program related to file transfer is stored, and a processor of the electronic device executes the computer program to perform file transfer management.

[0066] The following is a detailed description of the file transfer solution provided by the embodiment of the present application through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0067] As attached Figure 1 As shown, the file transmission method provided in the embodiment of the present application includes the following steps:

[0068] Step 101: When the file to be uploaded meets the breakpoint resume condition, the file is divided into multiple fragments according to the first preset rule to generate a first fragment set and an upload key is generated for the first fragment set. It should be noted that in the actual implementation process, when it is determined that there is a file to be uploaded, it is first determined whether the file to be uploaded needs to be uploaded in pieces; if it is not necessary to upload in pieces, the file transfer process shown in the embodiment of the present application is not executed; if it is necessary to upload in pieces, step 101 is executed to further determine whether the file to be uploaded meets the breakpoint resume condition.

[0069] The first preset rule can be flexibly set by the user, and there is no specific limitation on this in the embodiment of the present application. For example, the user can set the size of each fragment.

[0070] For example, if a file is 4.5 GB in size and the system sets the size of each fragment to 1 GB, the file can be divided into five fragments. The first four fragments are each 1 GB in size, and the fifth fragment is 0.5 GB in size. These five fragments constitute the first fragment set.

[0071] The system generates an attribute information for each shard, which can also be called parameter information. The shard parameter information can include but is not limited to: shard size, shard index, business identifier corresponding to the shard (also called upload key), and hash value of the shard. The hash value of the shard, also called MD5 value, is generated by encoding the summary content of the data in the shard through the hash algorithm.

[0072] The file to be uploaded can also be called the target object to be uploaded, wherein the historical upload information includes at least one shard set. For a file, when the upload is interrupted by different users or the same user through different terminals, a set of uploaded shards will be generated as part of the historical upload information. When each user requests to upload a file, the system assigns a service identifier (i.e., an upload key) to the user. Therefore, each shard set in the historical upload information corresponds to a service identifier. When the system receives a request from a user to upload a file, it first determines whether there is historical upload information of the file. If so, the breakpoint resume file transfer process shown in the embodiment of the present application is executed; if not, the file is initialized and fragmented, and multi-threaded synchronization is used to upload the data corresponding to each fragment.

[0073] In an optional embodiment, the method of determining whether the file to be uploaded meets the breakpoint resume condition may include the following steps:

[0074] Step 1: Determine the cache area allocated to the user;

[0075] The server can allocate a corresponding cache area for each user group, and a user group includes multiple users. The file fragment data uploaded by the user group is stored in the corresponding cache area. In addition, the cache area also stores the historical upload records of the file.

[0076] Step 2: Check whether there is any historical upload record of the file to be uploaded in the cache;

[0077] Step 3: If it exists, determine that the file to be uploaded meets the breakpoint resume condition and execute step 101; if it does not exist, determine that the file to be uploaded does not meet the breakpoint resume condition.

[0078] When it is determined that the file to be uploaded does not meet the breakpoint resume condition, it means that the user or the user group to which the user belongs has not uploaded the file in the past, the file is initialized to be fragmented and the data corresponding to each fragment is uploaded synchronously using multi-threading.

[0079] Step 102: Filter a set of historical shards that meet the conditions from the historical upload information of the file according to the upload key and the parameters of each shard in the first shard set.

[0080] The historical upload information may include multiple historical shard sets. In the embodiment of the present application, the historical shard sets that meet the conditions can be screened out through the upload key generated for the first shard set, that is, Key and shard parameters.

[0081] For example, if the file to be uploaded is file A, the system generates the first shard set for file A in sequence according to the first shard rule, and the shards with shard subscripts 0-M; the names of the shards in the historical shard set also include the shard subscripts. By comparing the historical shards with the shards in the first shard set with the same subscripts, the historical shard set that meets the conditions can be screened out. Step 103: According to the parameter information of each shard in the first shard set, a matching second shard set is searched from the historical shard set that meets the conditions.

[0082] Since a user or user group may try to upload a file multiple times through different terminals, but each attempt is interrupted during the upload process, there may be multiple historical shard sets that meet the conditions in the historical upload information. Accordingly, if a user or user group only uploads a file once and the upload process is interrupted, there may be only one historical shard set that meets the conditions in the historical upload information. Therefore, the historical upload information may contain one or more historical shard sets that meet the conditions.

[0083] When there is a historical shard set that meets the condition, the historical shard set can be determined as the second shard set. When there are at least two historical shard sets that meet the condition, it is necessary to search for a matching second shard set from the historical shard sets.

[0084] In an optional embodiment, according to the parameter information of each shard in the first shard set, a method of searching for a matching second shard set from a historical shard set that meets the condition may include the following sub-steps:

[0085] Sub-step 1: for each historical shard set included in each historical shard set that meets the conditions for historical message transmission, the parameter information of the shards in the historical shard set is matched with the parameter information of the shards in the first shard set one by one.

[0086] A feasible way of matching the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one for each historical shard set that meets the conditions may be as follows:

[0087] For each historical shard set that meets the conditions, extract the shard parameter information of a shard from the historical shard set; compare the shard subscript of the extracted shard with the shard subscript of the shard in the first shard set; if the two are the same, compare the hash values ​​of the shard pair. If the hash values ​​of the shard pair are the same, it is determined that the shard pair matches.

[0088] This method is used cyclically to determine the matching shards in the first shard set for each shard in a single historical shard set in each historical shard set that meets the conditions. Whenever a matching shard pair is polled, the historical shard set is increased by +1 until all shards in the historical shard set are polled to obtain the number of matching shards.

[0089] Sub-step 2: Determine the number of shards in the historical shard set that match the first shard set.

[0090] By looping through sub-steps 1 and 2, the number of shards corresponding to each historical shard set included in the historical shard set that meets the conditions can be determined.

[0091] Sub-step 3: Determine the historical shard set with the largest number of shards matching the first shard set as the second shard set.

[0092] The unique scoring algorithm provided in this optional embodiment can effectively identify the best set of fragments, reduce redundancy and invalid transmission, and effectively reduce the problem of fragment data confusion caused by multiple groups of fragments.

[0093] It should be noted that if the historical shard set that meets the conditions contains only one shard set, and no matching shard is determined through attribute matching with the shards in the first allocation set (that is, the number of shards in the shard set that match the first shard set is 0), the file breakpoint resume operation will not be performed, but the file will be initialized and sharded, and multi-threaded synchronous upload of the data corresponding to each shard will be used.

[0094] Step 104: Determine the different shards in the first shard set and the second shard set.

[0095] In the actual implementation process, the differential shards can be marked to facilitate subsequent breakpoint resumption of file transfers.

[0096] An optional way to determine the difference shards in the first shard set and the second shard set may be as follows:

[0097] Determine the shard pairs that match the first shard set and the second shard set; delete the shards in the second shard set that do not match the shards in the first shard set to obtain a third shard set; and determine each shard in the non-intersection of the first shard set and the third shard set as a differential shard.

[0098] Step 105: Use multiple threads to upload data corresponding to the differential shards, and merge the second shard set and the data corresponding to the differential shards into a target file.

[0099] The specific rules for merging target files can be flexibly set by those skilled in the art, and are not specifically limited in the embodiments of the present application. For example, the order of each fragment can be determined according to the fragment subscript of each fragment uploaded to the server, and the data corresponding to each fragment is sorted and merged according to the order. The upload key of the second set of fragments is used as the upload key of the file to be uploaded this time for continued transmission.

[0100] The file transmission method of the present application, when the file to be uploaded meets the breakpoint resume condition, divides the file into multiple fragments according to the first preset rule to generate a first fragment set and generates an upload key for the first fragment set; based on the upload key and the parameters of each fragment in the first fragment set, screens the historical fragment set that meets the condition from the historical upload information of the file; based on the parameter information of each fragment in the first fragment set, searches for a matching second fragment set from the historical fragment set that meets the condition; determines the difference fragments between the first fragment set and the second fragment set; uses multi-threading to upload the data corresponding to the difference fragments, and merges the data corresponding to the second fragment set and the difference fragments into the target file. The file transmission method provided by the present application uploads files by the breakpoint resume method, so that the fragments of the same file that have been uploaded in the past are used, which can not only avoid the problem of the fragments uploaded in the past occupying extra storage space, but also save the traffic resources consumed by file upload.

[0101] The following is a specific example to illustrate the file transmission method provided in the embodiment of the present application.

[0102] The file transfer method provided in this specific example supports breakpoint resumption by polling the uploaded fragments of the target object such as a file and grouping them (a group is a fragment set), identifying and scoring the matching fragments, thereby selecting the best group (fragment set), and continuing to resume the transmission based on the interruption of the group fragment, thereby reducing bandwidth consumption and waiting time for users and improving upload efficiency. The file transfer method of this specific example includes the following steps:

[0103] Step 1: Check whether there are any records of files to be uploaded that have not been uploaded yet.

[0104] Step 2: If the file has not been uploaded, initialize the multi-part upload; if the file has been uploaded previously, obtain the historical upload information.

[0105] Step 3: Get historical shard information and group the shards.

[0106] In this step, the fragment information of each historical upload is obtained, and the fragments can be grouped according to the business identifier corresponding to each fragment information, that is, the fragment group (also called fragment set) corresponding to each business identifier is obtained.

[0107] Step 4: If there are multiple groups of fragments, select the group with the highest matching degree; if there is only one group of fragments, directly calculate the matching score between the group and the group of fragments of the file to be uploaded.

[0108] For only one group of fragments, the matching score between the group and the group of fragments of the file to be uploaded can be calculated as follows:

[0109] The target historical fragment MD5, business identifier and fragment index are judged. If they are consistent with a fragment of the file to be uploaded, 1 point is accumulated for the group, otherwise no point is accumulated; poll each fragment in the group, calculate the score of the group, and use the group as the target group.

[0110] In the case where the historical upload information contains multiple groups of fragments, the above method is used to judge each group of fragments separately. Specifically, matching is performed based on information such as fragment size, fragment index, and fragment MD5. The matching score of each group with the fragment group of the file to be uploaded is determined respectively, and the group with the highest matching score is selected as the target group.

[0111] The target group is used as the data basis for subsequent file breakpoint resuming. It should be noted that if the match score of the target group is 0, the file is re-uploaded. If the match score of the target group is greater than 0, the uploaded historical fragments in the target group are used, and the unuploaded data fragments are marked.

[0112] Step 5: Use multiple threads to upload the unuploaded data segments.

[0113] Step 6: Merge the segments to be re-uploaded and the historical segments uploaded in the target group into a file.

[0114] It should be noted that the merging of shards is actually the merging of the data corresponding to the shards.

[0115] The file transfer method provided in this specific example can not only reduce the space occupied by junk segments by resuming the file transfer at the breakpoint, but also improve the file upload speed by utilizing the file segments uploaded in the past. Moreover, by utilizing the file segments uploaded in the past, data anomalies caused by erroneous segments can also be reduced.

[0116] Another specific file transmission method example is as follows, comprising the following steps:

[0117] Step 1: Determine the file A to be uploaded.

[0118] Step 2: Determine whether file A needs to be uploaded in parts; if so, proceed to step 3; if not, the process ends.

[0119] Step 3: Generate an upload key for file A.

[0120] In this step, if file A needs to be uploaded in pieces, file A is divided into multiple pieces to form a first piece set according to a first preset rule, and an upload key, also called a business identifier, is generated for the first piece set.

[0121] Step 4: Query the shard upload history information in the storage system according to the key, and filter out the historical shard set that meets the conditions based on the shard parameters.

[0122] Step 5: Determine whether the number of historical shard sets is greater than 0; if so, execute step 6; if not, re-upload the business ID to start uploading shards.

[0123] Step 6: Traverse the historical shard sets that meet the conditions and calculate the matching score between each historical shard set and the first shard set.

[0124] File A is divided into shards with subscripts 0-M in sequence according to the first preset rule. The shard name in the historical shard set contains the shard subscript. By comparing the hash values ​​of the local shard and the historical shard with the same subscript, it can be determined whether the two match. Whenever a matching shard pair is traversed, the matching score of the historical shard set corresponding to the shard pair is increased by 1. By traversing each shard in each historical shard set, the matching score corresponding to each historical shard set can be determined.

[0125] Step 7: After the traversal is completed, the historical shard set with the highest matching score is determined as the second shard set.

[0126] Step 8: Determine the different shards between the first shard set and the second shard set.

[0127] Step 9: Use multi-threading to upload the data corresponding to the differential shards, and merge the data corresponding to the second shard set and the differential shards into a target file.

[0128] It should be noted that when resuming the file transfer, the upload key of the second shard set is used as the upload key for the current file resuming transfer.

[0129] Figure 2 A structural block diagram of a file transmission device for implementing an embodiment of the present application.

[0130] The file transmission device provided in the embodiment of the present application includes the following functional modules:

[0131] The fragmentation module 201 is used for dividing the file to be uploaded into multiple fragments according to a first preset rule to generate a first fragment set and generate an upload key for the first fragment set when the file to be uploaded meets the breakpoint resume condition;

[0132] An information determination module 202 is used to select a historical shard set that meets the conditions from the historical upload information of the file according to the upload key and the parameters of each shard in the first shard set;

[0133] A search module 203 is used to search for a matching second shard set from the historical shard set that meets the conditions according to the parameter information of each shard in the first shard set;

[0134] A difference determination module 204, configured to determine different shards between the first shard set and the second shard set;

[0135] The resuming module 205 is used to upload the data corresponding to the differential shards by using multiple threads, and merge the second shard set and the data corresponding to the differential shards into a target file.

[0136] In some embodiments, the device further comprises:

[0137] A cache area determination module, used to determine the cache area allocated to the user;

[0138] A query module, used to query whether there is a historical upload record of the file to be uploaded in the cache area;

[0139] A query result determination module, used to determine if the file to be uploaded meets the breakpoint resume condition;

[0140] If it does not exist, it is determined that the file to be uploaded does not meet the breakpoint resume condition.

[0141] In some implementations, the shard parameter information includes: shard size, shard index, service identifier corresponding to the shard, and hash value of the shard.

[0142] In some implementations, the search module includes:

[0143] A first submodule is used to match the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one for each historical shard set that meets the condition;

[0144] A second submodule is used to determine the number of shards in the historical shard set that match the first shard set;

[0145] The third submodule is used to determine the historical shard set with the largest number of shards matching the first shard set as the second shard set.

[0146] In some embodiments, the first submodule includes:

[0147] The first unit is configured to extract, for each of the historical shard sets that meets a condition, shard parameter information of a shard from the shard set;

[0148] A second unit is used to compare the shard subscript of the shard with the shard subscripts of the shards in the first shard set;

[0149] The third unit is used for comparing the hash values ​​of the shard pairs if there are two identical shard pairs;

[0150] The fourth unit is configured to determine that the shard pair matches if the hash values ​​of the shard pair are the same. In some implementations, the difference determination module includes:

[0151] A fourth submodule, configured to determine matching shard pairs in the first shard set and the second shard set;

[0152] A fifth submodule, configured to delete the shards in the second shard set that are not matched with the shards in the first shard set, to obtain a third shard set;

[0153] The sixth submodule is configured to determine each shard in a non-intersection of the first shard set and the third shard set as a different shard.

[0154] The file transmission device of the present application, when the file to be uploaded meets the breakpoint resume condition, divides the file into multiple fragments according to the first preset rule to generate a first fragment set and generates an upload key for the first fragment set; based on the upload key and the parameters of each fragment in the first fragment set, screens the historical fragment set that meets the condition from the historical upload information of the file; based on the parameter information of each fragment in the first fragment set, searches for a matching second fragment set from the historical fragment set that meets the condition; determines the difference fragments between the first fragment set and the second fragment set; uses multi-threading to upload the data corresponding to the difference fragments, and merges the data corresponding to the second fragment set and the difference fragments into a target file. The file transmission device provided by the present application uploads files by the breakpoint resume method, so that the fragments of the same file that have been uploaded in the past are used, which can not only avoid the problem of the fragments uploaded in the past occupying extra storage space, but also save the traffic resources consumed by file upload.

[0155] It should be understood that the specific features, operations and details described hereinabove with respect to the method of the present application may also be similarly applied to the device and system of the present application, or vice versa. In addition, each step of the method of the present application described above may be performed by a corresponding component or unit of the device or system of the present application.

[0156] It should be understood that each module / unit of the device of the present application can be implemented in whole or in part by software, hardware, firmware or a combination thereof. Each module / unit can be embedded in the processor of the electronic device or independent of the processor in the form of hardware or firmware, or can be stored in the memory of the electronic device in the form of software for the processor to call to perform the operation of each module / unit. Each module / unit can be implemented as an independent component or module, or two or more modules / units can be implemented as a single component or module.

[0157] like Figure 3 As shown, the present application provides an electronic device 400, which includes a processor 401 and a memory 402 storing computer program instructions. The processor 401 implements the steps of the above-mentioned file transmission method when executing the computer program instructions. The electronic device 400 can be a server, a terminal, or any other electronic device with necessary computing and / or processing capabilities in a broad sense.

[0158] In one embodiment, the electronic device 400 may include a processor, a memory, a network interface, a communication interface, etc. connected through a system bus. The processor of the electronic device 400 may be used to provide necessary computing, processing and / or control capabilities. The memory of the electronic device 400 may include a non-volatile storage medium and an internal memory. The non-volatile storage medium may store an operating system, a computer program, etc. The internal memory may provide an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface and the communication interface of the electronic device 400 may be used to connect and communicate with external devices through a network. When the computer program is executed by the processor, the steps of the method of the present application are executed.

[0159] The present application provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned file transmission method is implemented.

[0160] Those skilled in the art will appreciate that the method steps of the present application can be completed by instructing related hardware such as the electronic device 400 or a processor through a computer program, and the computer program can be stored in a non-temporary computer-readable storage medium, and the computer program causes the steps of the present application to be executed when it is executed. Depending on the circumstances, any reference to memory, storage, or other media herein may include non-volatile or volatile memory. Examples of non-volatile memory include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, etc. Examples of volatile memory include random access memory (RAM), external cache memory, etc.

[0161] The various technical features described above can be combined arbitrarily. Although all possible combinations of these technical features are not described, any combination of these technical features should be considered to be covered by this specification as long as there is no contradiction in such combination.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A file transmission method, characterized in that: The method comprises: When the file to be uploaded meets the breakpoint resumable upload condition, the file is divided into a plurality of fragments according to a first preset rule to generate a first fragment set and an upload key is generated for the first fragment set; Filtering a historical shard set that meets the conditions from historical upload information of the file according to the upload key and the parameters of each shard in the first shard set; According to the parameter information of each shard in the first shard set, searching for a matching second shard set from the historical shard set that meets the conditions; Determine different shards in the first shard set and the second shard set; The data corresponding to the differential shards are uploaded using multiple threads, and the second shard set and the data corresponding to the differential shards are merged into a target file.

2. The method according to claim 1, characterized in that The method further comprises: Determine the buffer area allocated for the user; Querying whether there is a historical upload record of the file to be uploaded in the cache area; If so, determine that the file to be uploaded meets the breakpoint resume condition; If it does not exist, it is determined that the file to be uploaded does not meet the breakpoint resume condition.

3. The method according to claim 1, characterized in that The shard parameter information includes: shard size, shard index, service identifier corresponding to the shard, and hash value of the shard.

4. The method according to claim 3, characterized in that The step of searching for a matching second shard set from the historical shard set that meets the conditions according to the parameter information of each shard in the first shard set comprises: For each historical shard set that meets the condition, matching the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one; Determining the number of shards in the historical shard set that match the first shard set; A historical shard set having the largest number of shards matching the first shard set is determined as the second shard set.

5. The method according to claim 4, characterized in that For each historical shard set that meets the condition, the step of matching the parameter information of the shards in the historical shard set with the parameter information of the shards in the first shard set one by one includes: For each of the historical shard sets that meets the condition, extracting shard parameter information of a shard from the shard set; Comparing the shard subscript of the shard with the shard subscripts of the shards in the first shard set; If there are two identical ones, the hash values ​​of the shard pair are compared; If the hash values ​​of the shard pair are the same, then it is determined that the shard pair matches.

6. The method according to claim 1, characterized in that The step of determining the different shards in the first shard set and the second shard set includes: Determine matching shard pairs in the first shard set and the second shard set; Deleting the shards in the second shard set that are not matched with the shards in the first shard set to obtain a third shard set; Each shard in a non-intersection of the first shard set and the third shard set is determined as a differential shard.

7. A file transmission device, characterized in that: The device comprises: A fragmentation module, configured to, when a file to be uploaded meets a breakpoint-resume upload condition, divide the file into a plurality of fragments according to a first preset rule to generate a first fragment set and generate an upload key for the first fragment set; An information determination module, configured to filter a set of historical shards that meet conditions from historical upload information of the file according to the upload key and parameters of each shard in the first shard set; A search module, configured to search for a matching second shard set from the historical shard set that meets the conditions according to the parameter information of each shard in the first shard set; A difference determination module, configured to determine different shards between the first shard set and the second shard set; The resuming module is used to upload the data corresponding to the differential shards by using multiple threads, and merge the second shard set and the data corresponding to the differential shards into a target file.

8. The device according to claim 7, characterized in that The device also includes: A cache area determination module, used to determine the cache area allocated to the user; A query module, used to query whether there is a historical upload record of the file to be uploaded in the cache area; A query result determination module, used to determine if the file to be uploaded meets the breakpoint resume condition; If it does not exist, it is determined that the file to be uploaded does not meet the breakpoint resume condition.

9. An electronic device, characterized in that: The device comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the file transmission method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the file transmission method according to any one of claims 1 to 6 are implemented.

Images