A file uploading method, electronic device and computer readable medium

Abstract

The present application discloses a file uploading method, an electronic device and a computer-readable medium. The method comprises the following steps: determining a check value and a set of fragment summary values ​​of a file to be uploaded according to a preset check value determination method and a random data range pre-agreed with a server; sending a file start upload request message carrying the check value and the set of fragment summary values ​​to the server; uploading the file to be uploaded according to an upload task identifier and a collision detection condition carried in the incomplete upload notification message when an incomplete upload notification message sent by the server is received; parsing the complete upload notification message to obtain file storage information when a complete upload notification message sent by the server is received; the complete upload notification message is sent by the server when it is determined that a file with a check value exists locally and the set of fragment summary values ​​determined based on the file matches the set of fragment summary values ​​carried in the file start upload request message.

Description

Technical Field

[0001] The present application relates to the technical field of file uploading, and in particular, to a file uploading method, an electronic device, and a computer-readable medium. Background Art

[0002] File upload is an important function in cloud storage systems. Especially for small private clouds, it not only avoids uploading unnecessary files, but also ensures that only one copy of the same file is stored, thus avoiding wasted bandwidth and storage costs.

[0003] Current file upload methods are typically based on hash algorithms (such as MD5 and the SHA family). Specifically, the client first calculates the hash value of the file to be uploaded using the hash algorithm, then sends a data request containing this hash value to the server. The server then checks whether the hash value already exists, thereby determining whether it already has the file and returning the corresponding detection result to the client. However, this method has many drawbacks, such as long CPU resource usage, long latency, and low security. Summary of the Invention

[0004] The present application aims to solve one of the technical problems in the related art to a certain extent. To this end, the present application provides a file uploading method, an electronic device and a computer-readable medium.

[0005] As a first aspect of the present application, a file uploading method is provided for use on a client, wherein the method comprises:

[0006] Determine the checksum and fragment summary value set of the file to be uploaded based on the preset checksum determination method and the random data range pre-agreed with the server;

[0007] Sending a file upload start request message carrying the checksum and the fragment summary value set to the server;

[0008] Upon receiving an incomplete upload notification message sent by the server, uploading the file to be uploaded according to the upload task identifier and collision detection condition carried in the incomplete upload notification message; wherein the collision detection condition is generated by the server according to the set of fragment digest values ​​carried in the file upload start request message;

[0009] Upon receiving a complete upload notification message sent by the server, the complete upload notification message is parsed to obtain file storage information; wherein, the complete upload notification message is sent by the server when it is determined that a file with the verification value exists locally and a set of fragment summary values ​​determined based on the file matches the set of fragment summary values ​​carried in the file start upload request message.

[0010] Optionally, uploading the file to be uploaded according to the upload task identifier and the collision detection condition carried in the incomplete upload notification message includes:

[0011] If it is determined according to the collision detection condition that the collision detection fails, generating a file normal upload request message according to the file to be uploaded, and sending the file normal upload request message to the server;

[0012] If it is determined that the collision detection passes according to the collision detection condition, generating a file upload progress request message according to the upload task identifier, and sending the file upload progress request message to the server;

[0013] Receive a file upload progress response message sent by the server, and upload the file to be uploaded according to the file upload progress response message.

[0014] Optionally, uploading the file to be uploaded according to the file upload progress response message includes:

[0015] Obtain the number of uploaded bytes carried in the file upload progress response message;

[0016] Upload the data in the file to be uploaded to the server from the upload start point in the file to be uploaded that matches the number of bytes uploaded.

[0017] Optionally, determining the check value and the fragment summary value set of the file to be uploaded according to a preset check value determination method and a random data range pre-agreed with the server includes:

[0018] Dividing the file to be uploaded into a starting data block and a plurality of split data blocks according to the size of the file to be uploaded, a preset starting data block size, and a preset split data block size;

[0019] Determining a checksum of the file to be uploaded based on the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the plurality of split data blocks, and a preset checksum algorithm;

[0020] A set of fragments and a set of fragment summary values ​​thereof are determined based on a random data range pre-agreed with the server, the starting data block, and a plurality of segmented data blocks; wherein one fragment is one data block, and the fragment summary value corresponds one-to-one to the fragment.

[0021] Optionally, determining the checksum of the file to be uploaded based on the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the multiple split data blocks, and a preset checksum algorithm includes:

[0022] Determining the number of sample data block intervals according to the size of the file to be uploaded, the preset starting data block size, and the preset segmented data block size;

[0023] determining a plurality of sample data chunks from the plurality of segmented data chunks according to the number of sample data chunk intervals;

[0024] Determining a digest value of the file to be uploaded according to the size of the file to be uploaded, the starting data block, and the plurality of sample data blocks;

[0025] Determining a checksum of the digest value according to a preset checksum algorithm;

[0026] A checksum value of the file to be uploaded is determined based on the digest value and the checksum.

[0027] As a second aspect of the present application, a file uploading method is provided for use on a server, wherein the method comprises:

[0028] Receiving a file upload start request message sent by a client, and obtaining a checksum and a set of fragment digest values ​​of the file to be uploaded carried therein; wherein the checksum and the set of fragment digest values ​​are determined by the client according to a preset checksum determination method and a random data range pre-agreed with the server;

[0029] If it is determined that a file with the verification value exists locally and the set of fragment digest values ​​determined based on the file matches the set of fragment digest values ​​carried in the file upload start request message, sending a complete upload notification message carrying file storage information to the client;

[0030] When it is determined that the file to be uploaded is not completely uploaded, an incomplete upload notification message including an upload task identifier and a collision detection condition is sent to the client, so that the client can upload the file to be uploaded according to the upload task identifier and the collision detection condition; wherein the collision detection condition is generated by the server according to the set of fragment summary values ​​carried in the file start upload request message.

[0031] Optionally, when it is determined that the file to be uploaded is not completely uploaded, sending an incomplete upload notification message including an upload task identifier and a collision detection condition to the client includes:

[0032] In the case where it is determined that a file with the check value exists locally and a set of slice digest values ​​determined based on the file does not match the set of slice digest values ​​carried in the file upload start request message, and in the case where it is determined that a file with the check value does not exist locally and an unfinished task corresponding to the check value does not exist, an upload task identifier and a collision detection condition are generated, and an incomplete upload notification message carrying the upload task identifier and the collision detection condition is sent to the client;

[0033] When it is determined that there is no file with the verification value locally but there is an unfinished task corresponding to the verification value, a collision detection condition is generated, and an incomplete upload notification message carrying the upload task identifier of the unfinished task and the collision detection condition is sent to the client.

[0034] Optionally, when it is determined that a file having the verification value exists locally, the method further includes:

[0035] According to a random data range pre-agreed with the client, a set of fragments and a set of fragment summary values ​​thereof are determined from a plurality of data blocks corresponding to the locally existing file having the verification value; wherein one fragment is one data block, and the fragment summary value corresponds one-to-one to the fragment.

[0036] As a third aspect of the present application, an electronic device is provided, wherein the electronic device includes:

[0037] one or more processors;

[0038] A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the file upload method described in the first aspect of the present application or the file upload method described in the second aspect of the present application.

[0039] As a fourth aspect of the present application, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the file upload method described in the first aspect of the present application or the file upload method described in the second aspect of the present application is implemented.

[0040] In the file upload method provided in the embodiment of the present application, the client determines the check value and the fragment summary value set of the file to be uploaded according to a preset check value determination method and a random data range pre-agreed with the server, and sends a file start upload request message carrying the check value and the fragment summary value set to the server, which can quickly calculate the check value and the fragment summary value set used by the server to determine whether the file to be uploaded has been completely uploaded, effectively improving the file upload efficiency and avoiding waste of resources and time; subsequently, when the client receives an incomplete upload notification message sent by the server, the client uploads the file to be uploaded according to the upload task identifier and collision detection condition carried in the incomplete upload notification message, and the client performs collision detection on the collision detection condition generated by the server according to the fragment summary value set, which can further improve the file upload efficiency and security and avoid waste of resources and time. When the client receives a complete upload notification message sent by the server, it can directly obtain file storage information by parsing the complete upload notification message, without having to upload the file to be uploaded repeatedly, further avoiding waste of resources and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The present application will be further described below with reference to the accompanying drawings:

[0042] Figure 1 This is a flowchart of an implementation method of a file upload method executed by a client side provided in an embodiment of the present application;

[0043] Figure 2 This is a flowchart of an implementation method for uploading a file to be uploaded according to an upload task identifier and a collision detection condition provided in an embodiment of the present application;

[0044] Figure 3 This is a flowchart of an implementation method of uploading a file to be uploaded according to a file upload progress response message provided in an embodiment of the present application;

[0045] Figure 4 This is a flowchart of an implementation method of determining a checksum value and a fragment summary value set of a file to be uploaded provided in an embodiment of the present application;

[0046] Figure 5 This is a flowchart of an implementation method for determining the checksum of a file to be uploaded provided in an embodiment of the present application;

[0047] Figure 6 This is a schematic diagram of the division of files to be uploaded provided in an embodiment of the present application;

[0048] Figure 7 This is a flowchart of an implementation method of a file upload method executed by a server side provided in an embodiment of the present application;

[0049] Figure 8 This is a flowchart of an implementation method of sending an incomplete upload notification message including an upload task identifier and a collision detection condition to a client provided by an embodiment of the present application;

[0050] Figure 9 This is a module diagram of an implementation of an electronic device provided in an embodiment of the present application;

[0051] Figure 10 It is a schematic diagram of the computer-readable medium provided in an embodiment of the present application.

[0052] Description of Reference Numerals

[0053] 101: Processor 102: Memory

[0054] 103: I / O interface 104: bus DETAILED DESCRIPTION

[0055] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described in the embodiments are intended to be used to explain the present application and are not to be construed as limiting the present application.

[0056] References in this specification to "one embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with the embodiment itself can be included in at least one embodiment disclosed herein. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment.

[0057] Current file upload methods are typically based on hashing algorithms. However, the larger the file to be uploaded, the longer it takes to calculate the hash value. This increases the CPU usage and waiting time for the client, leading to increased resource and time waste. Furthermore, if a file's hash value remains a string of a specific length, it is theoretically possible to simulate the client and initiate an upload request to the server by enumerating all hash values, launching an attack on the server. It is even possible to retrieve the corresponding file based on the file representation returned by the server, leading to data leakage and seriously compromising data security.

[0058] In view of this, as a first aspect of the embodiment of the present application, a file uploading method is provided for use by a client, such as Figure 1 As shown, the method may include the following steps:

[0059] In step S110, the checksum and fragment summary value set of the file to be uploaded are determined according to the preset checksum determination method and the random data range pre-agreed with the server;

[0060] In step S120, a file upload start request message carrying the check value and the fragment summary value set is sent to the server;

[0061] In step S130, upon receiving the incomplete upload notification message sent by the server, uploading the file to be uploaded is performed according to the upload task identifier and collision detection condition carried in the incomplete upload notification message; wherein the collision detection condition is generated by the server according to the set of fragment digest values ​​carried in the file upload start request message;

[0062] In step S140, upon receiving the complete upload notification message sent by the server, the complete upload notification message is parsed to obtain file storage information; wherein, the complete upload notification message is sent by the server when it is determined that a file with the verification value exists locally and the set of fragment summary values ​​determined based on the file matches the set of fragment summary values ​​carried in the file start upload request message.

[0063] After receiving the file upload start request message from the client, the server determines whether the file requested by the client has been completely uploaded to the server based on the checksum value of the file to be uploaded and the set of fragment digest values ​​carried in the message. If the server determines that the file requested by the client has not been completely uploaded to the server, it sends an incomplete upload notification message to the client. If the server determines that the file requested by the client has been completely uploaded to the server, it sends a complete upload notification message to the client.

[0064] Upon receiving an incomplete upload notification message from the server, the client determines how to subsequently upload the pending file based on the upload task identifier and collision detection conditions carried in the incomplete upload notification message. Upon receiving a complete upload notification message from the server, the client parses the complete upload notification message to obtain the file storage information. It is understood that this file storage information records the relevant storage information of the pending file requested by the client after it was previously fully uploaded to the server.

[0065] In the file upload method provided in the embodiment of the present application, the client determines the check value and the fragment summary value set of the file to be uploaded according to a preset check value determination method and a random data range pre-agreed with the server, and sends a file start upload request message carrying the check value and the fragment summary value set to the server, which can quickly calculate the check value and the fragment summary value set used by the server to determine whether the file to be uploaded has been completely uploaded, effectively improving the file upload efficiency and avoiding waste of resources and time; subsequently, when the client receives an incomplete upload notification message sent by the server, the client uploads the file to be uploaded according to the upload task identifier and collision detection condition carried in the incomplete upload notification message, and the client performs collision detection on the collision detection condition generated by the server according to the fragment summary value set, which can further improve the file upload efficiency and security and avoid waste of resources and time. When the client receives a complete upload notification message sent by the server, it can directly obtain file storage information by parsing the complete upload notification message, without having to upload the file to be uploaded repeatedly, further avoiding waste of resources and time.

[0066] As described above, upon receiving the incomplete upload notification message sent by the server, the client will perform collision detection according to the collision detection conditions carried therein. Accordingly, in some embodiments, the file to be uploaded is uploaded (i.e., the file involved in step S130) according to the upload task identifier and collision detection conditions carried in the incomplete upload notification message, such as Figure 2 As shown, the following steps may be included:

[0067] In step S210, if it is determined that the collision detection fails according to the collision detection condition, a file normal upload request message is generated according to the file to be uploaded, and the file normal upload request message is sent to the server;

[0068] In step S220, when it is determined that the collision detection passes according to the collision detection condition, a file upload progress request message is generated according to the upload task identifier, and the file upload progress request message is sent to the server;

[0069] In step S230, a file upload progress response message sent by the server is received, and the file to be uploaded is uploaded according to the file upload progress response message.

[0070] For the case where the file requested to be uploaded by the client is not fully uploaded to the server, specifically, when the server determines that there is a file with the verification value locally and the set of fragment summary values ​​determined based on the file does not match the set of fragment summary values ​​carried in the file start upload request message, and when the server determines that there is no file with the verification value locally and there is no unfinished task corresponding to the verification value, the server generates an upload task identifier and a collision detection condition, and sends an incomplete upload notification message carrying the upload task identifier and the collision detection condition to the client. When the server determines that there is no file with the verification value locally but there is an unfinished task corresponding to the verification value, the server generates a collision detection condition, and sends an incomplete upload notification message carrying the upload task identifier of the unfinished task and the collision detection condition to the client.

[0071] In all three cases, the server determines that the file requested by the client has not been fully uploaded to the server. Therefore, the server sends an incomplete upload notification message to the client. However, in the first two cases, this indicates that the server currently does not have any unfinished tasks for the client, and the server needs to generate a new upload task identifier. In the last case, this indicates that the server currently has an unfinished task for the client and does not need to generate a new upload task identifier. Instead, the server directly returns the upload task identifier of the unfinished task to the client.

[0072] Regarding the collision detection condition, if the server determines that the file requested by the client has not been uploaded completely to the server, it will generate a collision detection condition based on the fragment digest value set received from the client and return it to the client. The client then performs a collision detection based on the collision detection condition. Specifically, the client compares the fragment digest value set in the collision detection condition with the fragment digest value set it has previously determined. If they are inconsistent, it means that the fragment digest value set sent by the client to the server has changed. If the uploaded file is also likely to be affected, the collision detection is considered to have failed. If they are consistent, the collision detection is considered to have passed.

[0073] When it is determined that the collision detection fails, the client will upload the file in a conventional hash algorithm upload manner, that is, generate a file normal upload request message according to the file to be uploaded, and send the file normal upload request message to the server.

[0074] When it is determined that the collision detection has passed, the client will perform an upload progress query, that is, generate a file upload progress request message according to the upload task identifier, and send the file upload progress request message to the server, receive the file upload progress response message sent by the server, and upload the file to be uploaded according to the file upload progress response message.

[0075] In some embodiments, the file to be uploaded is uploaded according to the file upload progress response message (ie, the step 230 involved), such as Figure 3 As shown, the following steps may be included:

[0076] In step S310, the number of uploaded bytes carried in the file upload progress response message is obtained;

[0077] In step S320, data in the file to be uploaded is uploaded to the server from the upload start point in the file to be uploaded that matches the number of bytes uploaded.

[0078] For example, if the number of uploaded bytes carried in the file upload progress response message is 0, the client will start uploading from the beginning of the file to be uploaded. If the number of uploaded bytes carried in the file upload progress response message is 50, the client will start uploading from the 51st byte of the file to be uploaded.

[0079] It can be understood that the entire file upload process involves several stages, namely, starting to request upload, querying upload progress, uploading each fragment data, and completing upload. The file upload method provided in the embodiment of the present application only optimizes the stage of starting to request upload, and therefore only involves the two stages of starting to request upload and querying upload progress, and does not elaborate on the subsequent stages of uploading each fragment data, completing upload, etc. In fact, starting from step S230, the client will execute the subsequent stages as usual.

[0080] The applicant of this application further proposes that the reason why the time required to calculate the hash value becomes longer as the file becomes larger in the related art is because the hash algorithm itself is relatively complex, and using the hash algorithm directly on a large file is even more complex. In view of this, the efficiency of calculating the checksum value can be greatly improved by first segmenting the file, then concatenating a specific data block with the file size as a digest value, and finally performing a checksum calculation on the digest value. Accordingly, in some embodiments, the checksum value of the file to be uploaded and the set of fragment digest values ​​are determined according to the preset checksum determination method and the random data range pre-agreed with the server (i.e., step S110), as shown in FIG. Figure 4 As shown, the following steps may be included:

[0081] In step S410, the file to be uploaded is divided into a starting data block and a plurality of split data blocks according to the size of the file to be uploaded, a preset starting data block size, and a preset split data block size;

[0082] In step S420, a checksum of the file to be uploaded is determined based on the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the multiple split data blocks, and a preset checksum algorithm;

[0083] In step S430, a set of shards and a set of shard summary values ​​thereof are determined based on the random data range pre-agreed with the server, the starting data block, and the plurality of split data blocks; wherein one shard is one data block, and the shard summary value corresponds one-to-one to the shard.

[0084] In the embodiment of the present application, there is no specific limitation on how to execute step S410. For example, a data block matching a preset starting data block size can be divided from the beginning of the file to be uploaded as the starting data block; then the remaining data in the file to be uploaded is divided into multiple data blocks according to the preset split data block size as split data blocks. When the length of the last data block is less than the preset split data block size, it is also regarded as a split data block.

[0085] After the client divides the file to be uploaded into a starting data block and multiple split data blocks, the file to be uploaded includes multiple continuous data blocks, from which multiple data blocks located in a random data range are selected, with one data block as a fragment, to form a fragment set, and the summary value of each fragment is calculated to form a fragment summary value set.

[0086] In the embodiment of the present application, there is no specific limitation on how to perform step S420. For example, the digest value of the file to be uploaded can be calculated first, and then the checksum of the digest value can be calculated and the checksum can be obtained by combining them. Accordingly, in some embodiments, the checksum of the file to be uploaded is determined based on the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the multiple split data blocks, and the preset checksum algorithm (i.e., step S420). Figure 5 As shown, the following steps may be included:

[0087] In step S510, the number of sample data block intervals is determined according to the size of the file to be uploaded, the preset starting data block size, and the preset split data block size;

[0088] In step S520, a plurality of specimen data chunks are determined from the plurality of segmented data chunks according to the number of specimen data chunk intervals;

[0089] In step S530, a digest value of the file to be uploaded is determined according to the size of the file to be uploaded, the starting data block, and the plurality of sample data blocks;

[0090] In step S540, a checksum of the digest value is determined according to a preset checksum algorithm;

[0091] In step S550, a checksum of the file to be uploaded is determined based on the digest value and the checksum.

[0092] In the embodiment of the present application, there is no specific limitation on how to execute step S510. For example, the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the number of split data blocks, the sample data size, the number of sample data blocks, and the number of sample data block intervals are represented as A, B, X, D, E, F, and G, respectively. Then, the total size of the multiple split data blocks is C = (AB), and the number of split data blocks is D = (C / X) = ((AB) / X). ​​The total size (AB) of the multiple split data blocks is divided by 10 to obtain the sample data size E = (AB) / 10. The sample data size E represents the total amount of data in the multiple split data blocks that participate in the summary value calculation. To limit the computational complexity, if the sample data size E obtained by dividing the total size of the multiple data chunks (AB) by 10 does not exceed a preset size (e.g., 200 MB), the calculated value of E can be maintained unchanged. If the sample data size E obtained by dividing the total size of the multiple data chunks (AB) by 10 does exceed the preset size (e.g., 200 MB), the calculated value of E can be set to the preset size (e.g., 200 MB). Divide the sample data size E by the preset data chunk size X to obtain the number of sample data chunks F = E / X. The number of sample data chunks F represents the total number of data chunks from the multiple data chunks that participate in the digest value calculation. That is, F data chunks of size X from the multiple data chunks participate in the digest value calculation. Divide the number of sample data chunks F by the number of sample data chunks D to obtain the number of sample data chunk gaps G. G represents the number of data chunks that separate two sample data chunks.

[0093] In the embodiment of the present application, there is no specific limitation on how to execute step S530. For example, the size of the file to be uploaded, the starting data block, and the multiple sample data blocks may be updated to the MD5 context, thereby calculating the MD5 value as the summary value of the uploaded file.

[0094] In the embodiment of the present application, there is no specific limitation on how to update multiple sample data blocks into the MD5 context. For example, the client can start a coroutine to read each sample data block and send it to a channel until multiple sample data blocks are sent, and then start another coroutine to read the sample data block from the channel and update it into the MD5 context until multiple sample data blocks are read and updated.

[0095] In the embodiments of the present application, the preset checksum algorithm is not specifically limited. For example, it can be a cyclic redundancy check (CRC32) algorithm. A fixed-length hexadecimal number can be formatted, and the checksum is filled in the hexadecimal number, with empty spaces filled with 0s.

[0096] In the embodiment of the present application, there is no specific limitation on how to execute step S550. For example, the checksum may be concatenated after the digest value to form a check value of a fixed length; or the checksum may be concatenated after the digest value and a fixed character string of a fixed number of bits may be padded before the digest value to form a check value of a fixed length.

[0097] The following reference Figure 6 For example, Figure 6 As shown, a schematic diagram of the division of the file to be uploaded provided in an embodiment of the present application is shown. First, a starting data block is divided from the file to be uploaded according to a preset starting data block size of n bytes. The starting data block is also the first fragment of the file to be uploaded (the fragment identifier can be represented by part_x, x can be an integer starting from 0, and x represents the (x+1)th fragment). Then, according to the preset split data block size m bytes, the remaining data in the file to be uploaded is divided into multiple split data blocks (the split data block identifier can be represented by block_x, x can be an integer starting from 0, and x represents the (x+1)th split data block). The first split data block is also the second fragment of the file to be uploaded. Finally, according to the sample data block interval number interval, multiple sample data blocks such as block_0 and block_n are selected from the multiple split data blocks. In addition, a fragment set v_part can also be selected and a fragment summary value set v_md5 can be calculated.

[0098] As a second aspect of the embodiment of the present application, a file uploading method is provided for use on a server, wherein: Figure 7 As shown, the method may include the following steps:

[0099] In step S610, a file upload start request message sent by a client is received, and a checksum and a set of fragment digest values ​​of the file to be uploaded carried in the message are obtained; wherein the checksum and the set of fragment digest values ​​are determined by the client according to a preset checksum determination method and a random data range pre-agreed with the server;

[0100] In step S620, if it is determined that a file with the verification value exists locally and the set of fragment digest values ​​determined based on the file matches the set of fragment digest values ​​carried in the file upload start request message, a complete upload notification message carrying the file storage information is sent to the client;

[0101] In step S630, when it is determined that the file to be uploaded is not uploaded completely, an incomplete upload notification message including an upload task identifier and a collision detection condition is sent to the client, so that the client can upload the file to be uploaded according to the upload task identifier and the collision detection condition; wherein, the collision detection condition is generated by the server according to the set of fragment summary values ​​carried in the file start upload request message.

[0102] The file upload method executed by the server side has been described above when describing the file upload method executed by the client side, so it will not be repeated here.

[0103] In the file upload method provided in the embodiment of the present application, the client determines the check value and the fragment digest value set of the file to be uploaded according to a preset check value determination method and a random data range pre-agreed with the server, and sends a file start upload request message carrying the check value and the fragment digest value set to the server. The check value and the fragment digest value set used by the server to determine whether the file to be uploaded has been completely uploaded can be quickly calculated, thereby effectively improving the file upload efficiency and avoiding waste of resources and time. Subsequently, when the server determines that there is a file with the check value locally and the fragment digest value set determined based on the file matches the fragment digest value set carried in the file start upload request message, the server sends a complete upload notification message carrying file storage information to the client, so that the client can directly obtain the file storage information by parsing the complete upload notification message, without having to repeatedly upload the file to be uploaded, further avoiding waste of resources and time. When the server determines that the file to be uploaded is not uploaded completely, it sends an incomplete upload notification message including an upload task identifier and a collision detection condition to the client, so that the client can upload the file to be uploaded according to the upload task identifier and the collision detection condition; wherein, the collision detection condition is generated by the server according to the set of fragment summary values ​​carried in the file start upload request message, so that the client performs collision detection according to the collision detection condition, which can further improve the efficiency and security of file uploading and avoid waste of resources and time.

[0104] In some embodiments, when it is determined that the file to be uploaded is not uploaded completely, an incomplete upload notification message (ie, involved in step S630) including an upload task identifier and a collision detection condition is sent to the client. Figure 8 As shown, the following steps may be included:

[0105] In step S710, when it is determined that a file with the check value exists locally and a set of slice digest values ​​determined based on the file does not match the set of slice digest values ​​carried in the file upload start request message, and when it is determined that a file with the check value does not exist locally and an unfinished task corresponding to the check value does not exist, an upload task identifier and a collision detection condition are generated, and an incomplete upload notification message carrying the upload task identifier and the collision detection condition is sent to the client;

[0106] In step S720, when it is determined that there is no file with the verification value locally but there is an unfinished task corresponding to the verification value, a collision detection condition is generated, and an incomplete upload notification message carrying the upload task identifier of the unfinished task and the collision detection condition is sent to the client.

[0107] As described above, when the server determines that a file with the verification value (i.e., the verification value sent by the client) exists locally, it further determines a set of fragment summary values ​​based on the file. Accordingly, in some embodiments, when it is determined that a file with the verification value exists locally, the method may further include the following steps: based on a random data range pre-agreed with the client, determining a set of fragments and a set of fragment summary values ​​thereof from a plurality of data blocks corresponding to the locally existing file with the verification value; wherein, one fragment is one data block, and the fragment summary value corresponds one-to-one to the fragment.

[0108] The following describes in detail the process of the file upload method provided by this application in conjunction with a most specific embodiment. The process of the file upload method can include the following steps:

[0109] S1. The client calculates the checksum K of the file to be uploaded;

[0110] Specifically, step S1 may include the following steps: the client updates the size A of the file to be uploaded into the MD5 context, ensuring that the size of the file to be uploaded participates in the calculation of the digest value of the file to be uploaded; the client divides a data block that matches a preset starting data block size B (for example, 1MB) from the beginning of the file to be uploaded as the starting data block, and also updates the starting data block into the MD5 context; the size of the remaining data in the file to be uploaded is C=(AB), and the client divides the remaining data of size C into a preset split data block size X (for example, 64KB) to obtain a number of split data blocks, D; the client divides C by 10 to obtain a sample data size E; the client divides the sample data size E by the preset split data block size X to obtain a number of sample data blocks F; The client divides the number of sample data blocks F by the number of split data blocks D to obtain the number of sample data block intervals G; determines multiple sample data blocks from the multiple split data blocks based on the number of sample data block intervals G, starts a coroutine to read each sample data block and sends it to a channel channel, until multiple sample data blocks are sent, starts another coroutine to read the sample data blocks from the channel channel and update them into the MD5 context, until multiple sample data blocks are read and updated; calculates a 32-bit MD5 value for the updated MD5 context; calculates the CRC32 checksum for the 32-bit MD5 value; concatenates the CRC32 checksum after the MD5 value, and pads a fixed string of fixed bits before the MD5 value to form a checksum K of fixed length.

[0111] S2. The client selects a set of shards within a random data range from the file to be uploaded and calculates a set of shard summary values.

[0112] S3. The client sends a file upload request message to the server, which includes a checksum K, a set of shards, and a set of shard summary values.

[0113] S4. The server receives the file upload start request message and obtains the check value K, the fragment set, and the fragment summary value set carried in the message;

[0114] S5. The server determines whether there is a file with the check value K (obtained in step S4) locally. If so, execute step S6; otherwise, execute step S6'.

[0115] S6. The server determines a set of fragments and a set of fragment summary values ​​from the multiple data blocks corresponding to the locally existing files with the check value K according to the random data range pre-agreed with the client; and proceeds to step S7.

[0116] S7. The server compares the fragment digest value set determined by itself with the fragment digest value set obtained in step S4. If they match, steps S8-S9 are executed; otherwise, steps S10-S11 are executed.

[0117] S8. The server sends a complete upload notification message carrying file storage information to the client;

[0118] S9. The client receives a complete upload notification message sent by the server, and parses the complete upload notification message to obtain file storage information;

[0119] S10: The server generates an upload task identifier and a collision detection condition, and sends an incomplete upload notification message carrying the upload task identifier and the collision detection condition to the client;

[0120] S11. The client receives an incomplete upload notification message sent by the server, and uploads the file to be uploaded according to the upload task identifier and collision detection condition carried in the incomplete upload notification message;

[0121] S6', the server determines whether there is an unfinished task corresponding to the check value K (obtained in step S4) locally. If so, execute step S7'; otherwise, jump to the above step S10;

[0122] S7', the server generates a collision detection condition, and sends an incomplete upload notification message carrying the upload task identifier of the unfinished task and the collision detection condition to the client; and jumps to the above step S11.

[0123] As a third aspect of the embodiments of the present application, an electronic device is provided, wherein, Figure 9As shown, the electronic device includes:

[0124] One or more processors 101;

[0125] The memory 102 stores one or more computer programs. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the file uploading method provided by the first aspect or the second aspect of the embodiment of the present application.

[0126] The electronic device may further include one or more I / O interfaces 103 connected between the processor 101 and the memory 102 and configured to implement information exchange between the processor 101 and the memory 102 .

[0127] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 102 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically such as SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) is connected between the processor and the memory, and can realize information exchange between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0128] In some embodiments, the processor 101 , the memory 102 , and the I / O interface 103 are connected to each other via a bus 104 , and further connected to other components of the computing device.

[0129] As a fourth aspect of the embodiment of the present application, Figure 10 As shown, a computer-readable medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, the file uploading method provided by the first aspect or the second aspect of the embodiment of the present application is implemented.

[0130] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, it can implement the method of any of the above-mentioned embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the embodiments of the present application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0131] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modifications that do not deviate from the functional and structural principles of the present application are included within the scope of the claims.

Claims

1. A file uploading method, used in a client, characterized in that: The method comprises: Determine the checksum and fragment summary value set of the file to be uploaded according to the preset checksum determination method and the random data range pre-agreed with the server; Sending a file upload start request message carrying the verification value and the fragment summary value set to the server; Upon receiving the incomplete upload notification message sent by the server, uploading the file to be uploaded according to the upload task identifier and the collision detection condition carried in the incomplete upload notification message; wherein the collision detection condition is generated by the server according to the set of fragment summary values ​​carried in the file start upload request message; When receiving a complete upload notification message sent by the server, parsing the complete upload notification message to obtain file storage information; wherein the complete upload notification message is sent by the server when it is determined that a file with the verification value exists locally and a set of fragment summary values ​​determined based on the file matches the set of fragment summary values ​​carried in the file start upload request message; Wherein, uploading the file to be uploaded according to the upload task identifier and the collision detection condition carried in the incomplete upload notification message includes: In the case that it is determined according to the collision detection condition that the collision detection fails, generating a file normal upload request message according to the file to be uploaded, and sending the file normal upload request message to the server; In the case where it is determined according to the collision detection condition that the collision detection passes, generating a file upload progress request message according to the upload task identifier, and sending the file upload progress request message to the server; Receive a file upload progress response message sent by the server, and upload the file to be uploaded according to the file upload progress response message.

2. The method according to claim 1, characterized in that The uploading of the file to be uploaded according to the file upload progress response message includes: Obtain the number of uploaded bytes carried in the file upload progress response message; Upload the data in the file to be uploaded to the server from the upload start point in the file to be uploaded that matches the number of bytes uploaded.

3. The method according to claim 1, characterized in that The method of determining the check value and the fragment summary value set of the file to be uploaded according to the preset check value determination method and the random data range pre-agreed with the server includes: Dividing the file to be uploaded into a starting data block and a plurality of split data blocks according to the size of the file to be uploaded, a preset starting data block size, and a preset split data block size; Determine the check value of the file to be uploaded according to the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the multiple split data blocks, and a preset check value algorithm; According to the random data range pre-agreed with the server, the starting data block and multiple segmented data blocks, a shard set and a shard summary value set thereof are determined; wherein one shard is one data block, and the shard summary value corresponds one-to-one to the shard.

4. The method according to claim 3, characterized in that The step of determining the checksum of the file to be uploaded according to the size of the file to be uploaded, the preset starting data block size, the preset split data block size, the starting data block and the multiple split data blocks, and a preset checksum algorithm includes: Determine the number of sample data block intervals according to the size of the file to be uploaded, the preset starting data block size, and the preset segmented data block size; Determine a plurality of sample data chunks from the plurality of segmented data chunks according to the number of sample data chunk intervals; Determining a summary value of the file to be uploaded according to the size of the file to be uploaded, the starting data block, and the multiple sample data blocks; Determine the checksum of the summary value according to a preset check value algorithm; A checksum value of the file to be uploaded is determined based on the digest value and the checksum.

5. A file uploading method, used in a server, characterized in that: The method comprises: Receive a file upload start request message sent by a client, and obtain a check value and a fragment summary value set of the file to be uploaded carried therein; wherein the check value and the fragment summary value set are determined by the client according to a preset check value determination method and a random data range pre-agreed with the server; When it is determined that a file having the verification value exists locally and a set of fragment digest values ​​determined based on the file matches a set of fragment digest values ​​carried in the file start upload request message, sending a complete upload notification message carrying file storage information to the client; When it is determined that the file to be uploaded is not completely uploaded, an incomplete upload notification message including an upload task identifier and a collision detection condition is sent to the client, so that the client can upload the file to be uploaded according to the upload task identifier and the collision detection condition; wherein the collision detection condition is generated by the server according to the set of fragment summary values ​​carried in the file start upload request message; Wherein, when it is determined that the file to be uploaded is not completely uploaded, sending an incomplete upload notification message including an upload task identifier and a collision detection condition to the client comprises: In the case where it is determined that a file with the check value exists locally and a set of shard summary values ​​determined based on the file does not match the set of shard summary values ​​carried in the file start upload request message, and in the case where it is determined that a file with the check value does not exist locally and there is no unfinished task corresponding to the check value, an upload task identifier and a collision detection condition are generated, and an incomplete upload notification message carrying the upload task identifier and the collision detection condition is sent to the client; When it is determined that there is no file with the verification value locally but there is an unfinished task corresponding to the verification value, a collision detection condition is generated, and an incomplete upload notification message carrying the upload task identifier of the unfinished task and the collision detection condition is sent to the client.

6. The method according to claim 5, characterized in that In the case where it is determined that a file having the verification value exists locally, the method further includes: According to a random data range pre-agreed with the client, a set of fragments and a set of fragment summary values ​​thereof are determined from a plurality of data blocks corresponding to the locally existing files having the verification value; wherein one fragment is one data block, and the fragment summary value corresponds one-to-one to the fragment.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors implement the file uploading method according to any one of claims 1-6.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the file uploading method according to any one of claims 1 to 6 is implemented.

Images