File processing method and apparatus, electronic device, and medium

Abstract

The present disclosure discloses a file processing method and device, electronic equipment and storage medium. The file processing method executed by an application node can include: dividing a target file to be transmitted into N file segments; wherein N is a positive integer greater than or equal to 2; determining a first hash value of each file segment; according to the first hash value, sending the N file segments to a second device; wherein the first hash value of the N file segments is used to combine to form a second hash value of the target file.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a document processing method and apparatus, electronic device and storage medium. Background Technology

[0002] The front-end (i.e., the file uploader) will first calculate the hash value of the file to be transmitted;

[0003] After the hash value is calculated, use the hash value as the input parameter to request the backend (i.e., the destination such as the file upload server);

[0004] Determine if the hash value already exists on the backend server;

[0005] If the file exists, check its upload status. If the upload is incomplete, receive the uploaded file from the front end.

[0006] If it does not exist, the hash value needs to be stored locally and the file uploaded by the front end needs to be received.

[0007] However, this file upload method can lead to long hash value calculation times for large files, resulting in significant file transfer delays. Summary of the Invention

[0008] This disclosure provides a file processing method and apparatus, electronic device and storage medium to simplify link tracing while reducing code improvements to application programs.

[0009] The first aspect of this disclosure provides a file processing method, executed by a first device, comprising:

[0010] The target file to be transmitted is divided into N file fragments; where N is a positive integer greater than or equal to 2.

[0011] Determine the first hash value of each of the aforementioned file fragments;

[0012] Based on the first hash value, the N file fragments are sent to the second device; wherein, the first hash value of the N file fragments is used to combine to form the second hash value of the target file.

[0013] Based on the above scheme, determining the first hash value of each file fragment includes:

[0014] Based on the position of the nth file fragment among the N file fragments, the sampling parameters for generating the first hash value of the nth file fragment are determined; wherein, n is a positive integer less than or equal to N;

[0015] Based on the sampling parameters, the first hash value of the nth file segment is obtained by sampling the nth file segment.

[0016] Based on the above scheme, the step of sampling the nth file segment according to the sampling parameters to obtain the first hash value of the nth file segment includes at least one of the following:

[0017] When n equals 1, the first hash value of the first file segment is obtained by sampling the first file segment;

[0018] When n equals N, the first hash value of the Nth file segment is obtained by sampling the Nth file segment;

[0019] When n is less than N and greater than 1, the remainder is obtained by taking the modulo of the first preset value of n, and the remainder is used to sample from one or more sub-segments of the nth file segment to obtain the first hash value of the nth file segment.

[0020] Based on the above scheme, when n is less than N and greater than 1, taking the modulo of n with a first preset value to obtain a remainder, and sampling from one or more sub-segments of the nth file segment according to the remainder to obtain the first hash value of the nth file segment, includes:

[0021] When the remainder is 0, the nth file segment is divided into sub-segments of the second preset value, and the sub-segments are sampled using a preset hash algorithm to obtain the first hash value of the nth file segment;

[0022] When the remainder is 1, the first S fields of the nth file segment are extracted using a preset hash algorithm as the first hash value of the nth file segment; where S is any positive integer less than the second preset value;

[0023] When the remainder is greater than or equal to 2, the nth file segment is divided into remainder sub-segments corresponding to the nth file segment using a preset hash algorithm, and one or more bytes are extracted from each sub-segment to obtain the first hash value of the nth file segment.

[0024] Based on the above scheme, determining the first hash value of each file fragment includes:

[0025] When uploading the file fragments, the first thread is used to determine the first hash value of each file fragment;

[0026] A second thread is used to determine a third hash value for the target file, wherein the third hash value is used to correct the second hash value; wherein the second thread is different from the first thread;

[0027] The third hash value is sent to the second device.

[0028] A second aspect of this disclosure provides a file processing method, characterized in that it is executed by a second device, the method comprising:

[0029] Receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided;

[0030] Determine whether the nth file fragment has been uploaded based on the first hash value;

[0031] When the nth file segment has not been uploaded, the nth file segment uploaded by the first device is received and cached.

[0032] Based on the above scheme, the method further includes:

[0033] After receiving and caching N file fragments, the second hash value of the target file is determined based on the first hash value of the first to Nth file fragments;

[0034] Based on the second hash value, it is determined whether to delete the cached target file or save the cached target file.

[0035] Based on the above scheme, determining whether to delete the cached target file or save the cached target file according to the second hash value includes at least one of the following:

[0036] If no fourth hash value matching the second hash value is found, the target file is saved.

[0037] When the fourth hash value, which is identical to the second hash value, is retrieved, it is determined that the cached target file will be deleted.

[0038] Based on the above scheme, the method further includes:

[0039] Receive the third hash value sent by the first device; the first thread is the thread within the first device that sends the file fragments and generates the first hash value;

[0040] Determine whether the second hash value and the third hash value are the same;

[0041] The step of determining whether to delete the cached target file or save the cached target file based on the second hash value includes:

[0042] When the second hash value and the third hash value are the same, the cached target file is either deleted or saved based on the second hash value.

[0043] A third aspect of this disclosure provides a document processing apparatus, comprising:

[0044] The file splitting module is used to divide the target file to be transmitted into N file fragments; where N is a positive integer greater than or equal to 2.

[0045] The first determining module is used to determine the first hash value of each of the file fragments;

[0046] The sending module is configured to send the N file fragments to the second device according to the first hash value; wherein the first hash value of the N file fragments is used to combine to form the second hash value of the target file.

[0047] A fourth aspect of this disclosure provides a document processing apparatus, the apparatus comprising:

[0048] The receiving module is used to receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided;

[0049] The second determining module is used to determine whether the nth file fragment has been uploaded based on the first hash value;

[0050] A receiving cache module is configured to receive and cache the nth file segment uploaded by the first device when the nth file segment has not been uploaded.

[0051] A fifth aspect of this disclosure provides an electronic device, the electronic device comprising:

[0052] Memory;

[0053] A processor, connected to the memory, is configured to implement the file processing method provided by either the first or second aspect by executing computer-executable instructions stored in the memory.

[0054] A sixth aspect of this disclosure provides a computer storage medium storing computer-executable instructions; when executed by a processor, the computer-executable instructions can implement the file processing method provided by any of the technical solutions of the first or second aspect.

[0055] The technical solution provided in this disclosure divides the target file into multiple file fragments, calculates the first hash value for each file fragment, and sends N file fragments to the second device sequentially based on the first hash value. Compared to first determining the second hash value of the entire target file and then sending the target file to the second device, the computation amount of a single first hash value is much smaller than that of the second hash value, which can reduce the delay at the start of the target file upload and improve the efficiency of the target file upload. Attached Figure Description

[0056] Figure 1 A flowchart illustrating a file processing method provided in an embodiment of this disclosure;

[0057] Figure 2 A flowchart illustrating a file processing method provided in an embodiment of this disclosure;

[0058] Figure 3 A flowchart illustrating a file processing method provided in an embodiment of this disclosure;

[0059] Figure 4 A flowchart illustrating a file processing method provided in an embodiment of this disclosure;

[0060] Figure 5 A flowchart illustrating a file processing method provided in an embodiment of this disclosure;

[0061] Figure 6 This is a schematic diagram of the structure of a document processing device provided in an embodiment of the present disclosure;

[0062] Figure 7 This is a schematic diagram of the structure of a document processing device provided in an embodiment of the present disclosure;

[0063] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0064] To gain a more detailed understanding of the features and technical content of this disclosure, the implementation of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit this disclosure.

[0065] like Figure 1 As shown, this disclosure provides a file processing method, executed by a first device, including:

[0066] S1110: Divide the target file to be transmitted into N file fragments; wherein, N is a positive integer greater than or equal to 2;

[0067] S1120: Determine the first hash value of each of the file fragments;

[0068] S1130: Based on the first hash value, send the N file fragments to the second device; wherein, the first hash value of the N file fragments is used to combine to form the second hash value of the target file.

[0069] The first device can be any upload device that needs to upload the target file to the second device. For example, the first device can be any terminal device, which can be a front end.

[0070] The second device may be a server; for example, the server may be a cloud server or a database server.

[0071] The target file can be: a text file, a video file, a mixed file of multiple media types, etc.

[0072] For example, if the data volume of the target file is greater than a set value, S1110 is executed; otherwise, the second hash value of the entire target file can be calculated directly, and the target file can be transmitted directly based on the second hash value.

[0073] In some embodiments, S1110 may include:

[0074] The target file is divided into N equal parts, resulting in N file fragments;

[0075] or,

[0076] The target file is divided into N file fragments based on a specific data value. In this way, the data size of the last file fragment can be less than or equal to the specific data value.

[0077] or,

[0078] Based on different chapters or content relevance of the target content, the target file is divided into N file fragments. Thus, the data size of each file fragment can be different, but not greater than a specific data value.

[0079] Of course, the above is just an example of dividing the target file into N file fragments, and the specific implementation is not limited to the above example.

[0080] After dividing the file into N file fragments, the first device will calculate the first hash value of each file fragment one by one. Since the file fragments are formed from the target file, the data volume of a single file fragment (or simply fragment) is small. When calculating the first hash value by sampling, the amount of data is much smaller than that of the entire file. This greatly advances the start time of transmission for target files with large data volumes, thereby improving transmission efficiency.

[0081] S1120 may include: sampling the content of each file fragment using various hash algorithms (e.g., hash algorithms or MD5 algorithms) to obtain the first hash value. There are various ways to calculate the first hash value, and the specific implementation is not limited to the examples above.

[0082] For example, S1120 may include: sampling part or all of the content of each file fragment to obtain a first hash value.

[0083] In some embodiments, S1130 may include:

[0084] Send a request message to the second device with the first hash value as the input parameter of the upload request;

[0085] The system receives a response message from the second device based on the request message. This response message determines whether the corresponding file fragment has been uploaded based on whether the second device has already stored the first hash value.

[0086] If the file fragment corresponding to the response message has not been uploaded, send the file fragment to the second device.

[0087] For example, the request message may contain one or more first hash values. For instance, if multiple hash values ​​are included, multiple file fragments can be uploaded simultaneously, improving upload efficiency. For example, the first solder values ​​corresponding to multiple consecutive file fragments in the target file are sequentially carried in the request message and sent to the second device.

[0088] As another example, the request message may include a first hash value, for instance, the request message being sent to the second device sequentially according to the order of file fragments in the target file.

[0089] The first device transmits the first hash value generated for each file fragment to the second device. The second device generates a second hash value based on the received first hash value. This second hash value can be used by the second device to verify whether the second device has accepted the upload of the target file.

[0090] like Figure 2 As shown, this disclosure provides a file processing method, executed by a first device, including:

[0091] S1210: Divide the target file to be transmitted into N file fragments; wherein, N is a positive integer greater than or equal to 2;

[0092] S1220: Based on the position of the nth file fragment among the N file fragments, determine the sampling parameters for generating the first hash value of the nth file fragment; wherein, n is a positive integer less than or equal to N;

[0093] S1230: Based on the sampling parameters, sample the nth file segment to obtain the first hash value of the nth file segment.

[0094] S1240: Based on the first hash value, send the N file fragments to the second device; wherein, the first hash value of the N file fragments is used to combine to form the second hash value of the target file.

[0095] In this embodiment of the disclosure, to further reduce the computational cost required to determine a single first hash value and to further reduce the latency introduced by the first hash value, a sampling parameter is determined based on the position of each file fragment in the target file (i.e., the position of the nth file fragment among the N file fragments). This sampling parameter, combined with the hash algorithm, can generate the first hash value.

[0096] The sampling parameters include, but are not limited to, at least one of the following:

[0097] The amount of data sampled in a file fragment;

[0098] The content of the corresponding file fragment to be sampled, which may be part or all of the content of the corresponding file fragment;

[0099] The amount of sampled data used to form the first hash value.

[0100] Of course, the above are just examples of sampling parameters, and the actual implementation is not limited to the examples above.

[0101] After obtaining the sampling parameters, the first hash value is obtained by sampling each file fragment according to the sampling parameters and the hash algorithm.

[0102] In some embodiments, when segmenting the target file, each file segment is sequentially numbered according to the file's order from front to back. Thus, n can be the number of the file segment.

[0103] For example, the step of sampling the nth file fragment according to the sampling parameters to obtain the first hash value of the nth file fragment includes at least one of the following:

[0104] When n equals 1, the first hash value of the first file segment is obtained by sampling the first file segment;

[0105] When n equals N, the first hash value of the Nth file segment is obtained by sampling the Nth file segment;

[0106] When n is less than N and greater than 1, the remainder is obtained by taking the modulo of the first preset value of n, and the remainder is used to sample from one or more sub-segments of the nth file segment to obtain the first hash value of the nth file segment.

[0107] When processing the first hash value of the first file fragment, the first thread assigned to file upload on the first device may not have any other processing tasks, meaning the first device is in a relatively idle state. Therefore, sampling the entire fragment of the first file fragment to obtain the first hash value is fast and efficient.

[0108] In some embodiments, the last file segment has a relatively small data volume or is smaller than the data volume of other file segments. In this case, sampling the last file segment (i.e., the Nth file segment) to obtain the first hash value is computationally inefficient and cost-effective. Therefore, in this embodiment of the present disclosure, both the first and Nth file segments can sample the entire file segment to obtain the first hash value.

[0109] In some embodiments, a file segment for sampling of the entire file segment can be selected based on a preset sequence of values, and a first hash value can be calculated.

[0110] In this embodiment, if n is between 1 and N, the first hash value of the corresponding file segment is obtained by taking the remainder of n modulo a first preset value. For example, when the remainder is 0, a specific sequence can be set as the first hash value of the file segment. For example, when the preset value is not 0, the corresponding file segment is divided into sub-segments equal to the remainder, and the first hash value of the file segment is obtained by sampling the first sub-segment, the last sub-segment, or a sub-segment at a specific position. Of course, the above are just examples of the first hash value, and the specific implementation is not limited to the above example values.

[0111] In some embodiments, when n is less than N and greater than 1, taking the modulo of n with a first preset value to obtain a remainder, and sampling from one or more sub-segments of the nth file segment based on the remainder to obtain the first hash value of the nth file segment, includes:

[0112] When the remainder is 0, the nth file segment is divided into sub-segments of the second preset value, and the sub-segments are sampled using a preset hash algorithm to obtain the first hash value of the nth file segment;

[0113] When the remainder is 1, the first S fields of the nth file segment are extracted using a preset hash algorithm as the first hash value of the nth file segment; where S is any positive integer less than the second preset value;

[0114] When the remainder is greater than or equal to 2, the nth file segment is divided into remainder sub-segments corresponding to the nth file segment using a preset hash algorithm, and one or more bytes are extracted from each sub-segment to obtain the first hash value of the nth file segment.

[0115] For example, if the remainder is 0, the file is divided into a second preset number of sub-segments, and samples are taken from the first one or more bytes of each sub-segment or the last one or more bytes of each sub-segment to obtain the first hash value. In summary, if the preset value is 0, the first hash value is obtained by sampling bytes at specific positions in each sub-segment.

[0116] When the default value is 1, after the corresponding file is divided into multiple sub-parts, S can be either 1 or 2.

[0117] When the remainder is greater than or equal to 2, the nth file fragment is divided into several sub-fragments corresponding to the remainder of the nth file fragment using a preset hash algorithm, and one or more bytes are extracted from each sub-fragment to obtain the first hash value of the nth file fragment, which may include:

[0118] After dividing the file fragments into sub-fragments based on the remainder, one or more bytes are further extracted from each sub-fragment (for example, the number of bytes in each sub-fragment is determined according to a preset sequence) to obtain the first hash value.

[0119] For example, the preset sequence may exhibit at least one of the following trends:

[0120] The sequence of elements tends to be large at both ends and small in the middle;

[0121] The elements of the sequence tend to be smaller at the beginning and larger at the end.

[0122] In some embodiments, determining the first hash value of each of the file fragments includes:

[0123] When uploading the file fragments, the first thread is used to determine the first hash value of each file fragment;

[0124] The method further includes:

[0125] A second thread is used to determine a third hash value for the target file, wherein the third hash value is used to correct the second hash value; wherein the second thread is different from the first thread; and the third hash value is sent to the second device.

[0126] For example, after generating the first hash value, the first thread concatenates multiple first hash values ​​according to the sorting position of the file fragment corresponding to the first hash value among N file fragments to obtain the second hash value. Of course, this is just one way to generate the second hash value based on the first hash value.

[0127] In other examples, a preset value can be inserted between two adjacent first hash values, and then the second hash value can be generated.

[0128] The second thread here differs from the first thread. The first thread can be the main thread for uploading the target file, and may include: calculating and processing the first hash value, encoding and decoding information exchanged between the second device, and processing the second hash value, etc.

[0129] The second thread can be another thread set up in parallel with the first thread, and its thread resources can be slightly less than those of the first thread. This second thread can be used to calculate a third hash value based on the file content of the target file. The method by which the second thread generates the third hash value can be the same as the method by which the first thread generates the first hash values ​​of multiple file fragments and then combines them to obtain the second thread's result. The third hash value generated in this way can then be used to correct the second hash value.

[0130] Since the second thread is independent of the first thread, the process of calculating the third hash value by the second thread will not interfere with the first thread's upload processing of the target file, thus ensuring the efficiency of the first thread's target content processing. For example, this second thread may include, but is not limited to, a Web Worker thread.

[0131] After generating the third hash value, the second thread provides it to the first thread. Upon receiving the third hash value, the first thread provides it to the server as a file upload request message or other type of message. This third hash value can be used by the second device to verify the second hash value generated based on the first hash value.

[0132] like Figure 3 As shown, this disclosure provides a file processing method, executed by a second device, the method comprising:

[0133] S2110: Receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided;

[0134] S2120: Determine whether the nth file fragment has been uploaded based on the first hash value;

[0135] S2130: When the nth file segment has not been uploaded, receive and cache the nth file segment uploaded by the first device.

[0136] In this embodiment, the second device may be a server, such as a cloud storage server or a database server.

[0137] The second device can be any type of server, such as a blade server.

[0138] The second device receives file fragments and a first hash value from the first device. This first hash value may be carried in the upload request. An upload request may include the first hash values ​​of one or more file fragments.

[0139] Upon receiving the first hash value, the second device compares the received first hash value with the hash value already stored locally. Based on the comparison result, it determines whether the second device has already stored the first hash value locally. If the first hash value has been stored locally, it means that the corresponding file fragment has been uploaded. If the first hash value has not been stored locally, it means that the file fragment corresponding to the first hash value has not been stored locally.

[0140] If the nth file fragment is not stored, the response message of the request message is returned to the first device, and the first device will upload the corresponding nth file fragment according to this device.

[0141] If a request message has multiple first hash values, some of the first hash values ​​may have been stored locally on the second device, while others may not have been stored on the second device. In this case, the second device can return the first hash values ​​that are not stored locally to the first device. The first device will then determine the file fragments to be uploaded to the second device based on the first hash values ​​carried in the response message, and upload the corresponding file fragments to the second device.

[0142] Of course, the above is just an example of file fragmentation. In this embodiment of the disclosure, since the target file is uploaded based on the first hash value of the file fragment, the file fragmentation upload efficiency is high.

[0143] In some embodiments, the method further includes:

[0144] After completing N first hash values, the second hash value of the target file is determined based on the first hash values ​​of the 1st to Nth file fragments;

[0145] Based on the second hash value, it is determined whether to delete the cached target file or save the cached target file.

[0146] After receiving and buffering N file fragments, the second device generates a second hash value for the entire target file based on the received N first hash values. For example, the second hash value is obtained by concatenating the N first hash values ​​according to their positions in the target file.

[0147] For example, N is carried in the first upload request (or the first request message for uploading the target file) sent by the first device. Thus, the second device can determine whether it has received N first hash values ​​from the first device based on its own count. If it receives N first hash values, it can then initiate the generation of the second hash value.

[0148] While receiving the first hash value, the first device is also uploading various file fragments. After receiving the file fragments, the second device first caches them locally. After generating the second hash value, it determines whether the target file has already been received and stored based on the second hash value. If it has been stored, it returns a stop upload instruction to the first device. Thus, upon receiving the stop upload instruction, the first device stops uploading any remaining unfinished file fragments.

[0149] If the second hash value determines that the second device has not received the target file, the cached target file will be converted into actual storage. If the second hash value determines that the second device has already stored the target file, one or more file fragments of the cached target file will be deleted.

[0150] In some embodiments, determining whether to delete the cached target file or save the cached target file based on the second hash value includes at least one of the following:

[0151] If no fourth hash value matching the second hash value is found, the target file is saved.

[0152] When the fourth hash value, which is identical to the second hash value, is retrieved, it is determined that the cached target file will be deleted.

[0153] In the embodiments of this disclosure, any hash value stored in the second device can be referred to as a fourth hash value. If no fourth hash value matching the second hash value is found, the target file can be at least temporarily saved; otherwise, the cached target file can be deleted.

[0154] The target file cached here is a temporary file of the target file. If it is determined that the target file will be saved, the temporary file will be converted into a real file for storage, for example, the target file will be stored in a specific path.

[0155] like Figure 4 As shown, this disclosure provides a file processing method, executed by a second device, the method comprising:

[0156] S2210: Receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided;

[0157] S2220: Determine whether the nth file fragment has been uploaded based on the first hash value;

[0158] S2230: When the nth file segment has not been uploaded, receive and cache the nth file segment uploaded by the first device;

[0159] S2240: Receive the third hash value sent by the first device; the first thread is the thread within the first device that sends the file fragments and generates the first hash value;

[0160] S2250: Determine whether the second hash value and the third hash value are the same;

[0161] S2260: When the second hash value and the third hash value are the same, determine whether to delete the cached target file or save the cached target file based on the second hash value.

[0162] In one embodiment, such as Figure 4 As shown, the method further includes:

[0163] S2270: When the second hash value and the third hash value are different, determine whether to delete the cached target file or save the cached target file based on the third hash value.

[0164] The second thread of the first device generates a third hash value for the target file. After generating the third hash value, the first device submits it to the second device in various ways as soon as possible. In this way, the second device can immediately verify whether the second hash value generated based on the first hash value is correct.

[0165] If the second hash value and the fourth hash value are to be compared, in order to ensure accuracy, the generated second hash value and the third hash value need to be consistent. If they are inconsistent, the third hash value and the fourth hash value are directly compared.

[0166] For example, if the third hash value is received after the second hash value is generated, the second hash value and the fourth hash value can be compared directly. After receiving the third hash value, the second hash value and the third hash value can then be compared again. This allows the first and second devices to continue transmitting the remaining file fragments based on the first and second hash values. If the third hash value is obtained before the second hash value is generated, the generated second hash value and the third hash value can be compared. If they match, the third hash value or the second hash value can be compared with the fourth hash value, respectively.

[0167] In some embodiments, the accuracy of the third hash value may be higher than the accuracy of the second hash value generated by the second device based on the first hash value. After receiving the third hash value, it can be directly compared with the locally stored fourth hash value. If the compared fourth hash value contains the third hash value, the transmission of file fragments of the target file is stopped, and the cached file fragments of the target file are discarded.

[0168] When uploading large files, calculating the file hash value is a time-consuming operation. Taking a typical browser upload as an example, for a 1.11GB .zip compressed file, calculating the hash value 10 times takes an average of 11.694 seconds. If the file is even larger, calculating the file hash will take even longer, which will affect the overall upload efficiency and reduce the user experience.

[0169] This disclosure proposes a method for uploading large files with fast front-end processing, which mainly consists of three parts: file sampling hash value calculation, an optional compensation mechanism that uses Web Worker for full hash calculation, and file slice uploading. It supports breakpoint resumption and instant uploading.

[0170] Based on user selection, the front-end determines the large file to be uploaded;

[0171] Based on user input in the front-end interface, the sampling ratio for file hash calculation is determined. For example, for intuitive quantification, the sampling ratio unit here is rB / 1Mb (meaning r bytes are extracted from every 1MB file).

[0172] The front-end performs sampling hash value calculation on large files to obtain the sampled hash value of the file. Here, the front-end is equivalent to the first device mentioned earlier.

[0173] The sampling hash algorithm of this disclosure is described as follows (taking a sampling ratio of 2B / 1MB as an example):

[0174] The file is divided into 5MB segments, and each segment is marked with an index starting from 0.

[0175] The first and last file segments are fully extracted, and the remaining file segments are sampled and truncated.

[0176] The algorithm calculates the remainder when the file's fragment identifier (index) is divided by 5. If the remainder is 0, the fragment is divided into 5 equal parts, each 1MB in size, and the first 2 bytes of each part are extracted. If the remainder is 1, the first 10 bytes of the fragment are extracted. If the remainder is 2, the fragment is divided into two parts, 3MB and 2MB in size, and the first 5 bytes of each part are extracted. If the remainder is 3, the fragment is divided into three parts, 2MB, 1MB, and 2MB in size, and 4, 2, and 4 bytes are extracted respectively. If the remainder is 4, the fragment is divided into four parts, 1MB, 1MB, and 2MB in size, and 2, 2, 2, and 4 bytes are extracted respectively. The fragment identifier can be the fragment number.

[0177] After the above steps, a final list of sampled files is obtained. A hash value is calculated for each sampled file, and then the hash values ​​are merged to obtain the final hash value for that file. For example, the spark-md5 third-party library can be used, which has the function of calculating file hash values ​​in chunks and merging hash values.

[0178] Using this method to calculate the hash value of a file, the average time for 10 hash calculations is 0.372 seconds. Compared to the average time of 11.694 seconds for calculating the hash of the entire file, this sampling algorithm improves the efficiency of hash calculation by approximately 31.44 times.

[0179] After receiving the hash value, the backend server checks its database to see if it already exists. If it doesn't, it indicates the file is being uploaded for the first time. The server then stores the hash value and notifies the frontend to begin the file upload. If the hash value exists, it means the file has been uploaded previously. The server then checks the upload status of the file. If it's 1 (indicating the file has been uploaded successfully), the server directly notifies the frontend that the upload is complete (instant upload). If it's 0 (indicating the file hasn't been uploaded successfully), the server returns a list of received file fragment identifiers to the frontend. The frontend then compares these and begins uploading the file fragments that haven't yet been received (resumable upload). The backend server here can be the second device mentioned earlier.

[0180] refer to Figure 5 As shown, the specific steps for uploading files on the front end are as follows:

[0181] The large file fragment list is obtained during the sampling hash calculation. Assuming there are 1000 fragment files, each file is labeled (from 1 to 1000). To avoid excessive concurrent requests leading to memory overflow, the concurrency is controlled at 5. That is, initially, 5 concurrent requests are sent to upload file fragments labeled 1, 2, 3, 4, and 5. After receiving each response, a new request is sent, uploading file fragments labeled 6, 7, and so on in sequence. Upon receiving the file fragments, the backend places these temporary file fragments into a cache and stores the received file fragment labels in the database for later retrieval during resume uploads.

[0182] Once the frontend has uploaded all file fragments and confirmed receipt of the response, it sends a request to the backend server to inform it that all file fragments have been uploaded. The backend server then begins merging the files, deleting temporary files to release the cache, and setting the upload status of the file to 1 in the database (indicating that the file has been uploaded successfully). After the file merging is complete, it responds to the frontend request, informing it that the entire file has been uploaded. This concludes the entire large file upload process.

[0183] If the compensation mechanism is enabled based on user actions:

[0184] After the compensation mechanism is activated, a Web Worker thread will be started to perform the full hash calculation task. The Web Worker thread creates a multi-threaded environment for JavaScript, allowing the main thread to create worker threads. The worker thread executes independently of the main thread in the background and can return the execution result to the main thread after it is completed.

[0185] While the full hash value is being calculated (which can be the aforementioned third hash value), the file upload proceeds normally.

[0186] Once the full hash of the file is calculated, the hash value is returned to the main thread. The main thread then calls the backend interface to query whether the hash already exists on the backend server. If it already exists and the upload is complete, the frontend is notified to stop the file upload and clear the temporary files.

[0187] If the file already exists but has not been completely uploaded, the file fragments that have been received by the frontend are returned, and the frontend begins to execute the subsequent steps of resuming the upload.

[0188] If the file does not exist, it indicates that this is the first time the file has been uploaded. No additional action is required; the file upload process will proceed as usual.

[0189] It is worth noting that the aforementioned second hash value is also the full hash value of the uploaded file.

[0190] This disclosure mainly proposes the concept of large file sampling hash value calculation. The application scenario is uploading large files through a browser page. It can quickly calculate the hash value of the file, determine the file upload status, save the time loss caused by the hash value calculation process of large files, and improve upload efficiency.

[0191] In summary, after the large file is fragmented, data is sampled from each fragment to obtain a list of sampled file fragments. After hash calculation is performed on each sampled file fragment, these hash values ​​are merged to obtain the sampled hash value of the large file as a unique identifier for the file.

[0192] This disclosure also proposes an optional compensation mechanism that uses the browser's Web Worker to simulate a multi-threaded execution environment for single-threaded JavaScript. The worker child thread calculates the full hash (i.e., calculates the full hash value of a large file in the background while uploading the file) without affecting the execution of the main thread, ensuring the accuracy and precision of the hash value.

[0193] Figure 5 The backend server returns a backend interface, which can be understood as the backend server telling the frontend whether to continue uploading the corresponding file fragments. This backend interface can be understood as a query interface.

[0194] like Figure 6 As shown, this disclosure provides a document processing apparatus, including:

[0195] The segmentation module 110 is used to divide the target file to be transmitted into N file fragments; wherein, N is a positive integer greater than or equal to 2;

[0196] The first determining module 120 is used to determine the first hash value of each of the file fragments;

[0197] The sending module 130 is used to send the N file fragments to the second device according to the first hash value; wherein the first hash value of the N file fragments is used to combine to form the second hash value of the target file.

[0198] In some embodiments, the file processing apparatus may be a first device.

[0199] In one embodiment, the segmentation module 110 and the first determining module 120 may correspond to a processor. The transmitting module 130 may correspond to a network interface or transceiver antenna, etc. The processor may be a central processing unit, microprocessor, digital signal processor, or embedded processor, etc.

[0200] In another embodiment, the division module 110, the first determination module 120, and the sending module 130 may be program modules; the program modules, after being executed by the processor, can perform the above operations.

[0201] In some embodiments, the division module 110, the first determination module 120, and the sending module 130 may be hardware-software combined modules; the hardware-software combined modules include, but are not limited to, programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0202] In some embodiments, the division module 110, the first determining module 120, and the sending module 130 may be pure hardware modules; the pure hardware modules include, but are not limited to, various application-specific integrated circuits.

[0203] In one embodiment, the first determining module 120 is specifically configured to determine sampling parameters for generating the first hash value of the nth file segment based on the position of the nth file segment among the N file segments; wherein, n is a positive integer less than or equal to N; and to sample the nth file segment according to the sampling parameters to obtain the first hash value of the nth file segment.

[0204] In one embodiment, the first determining module 120 is specifically configured to perform at least one of the following:

[0205] When n equals 1, the first hash value of the first file segment is obtained by sampling the first file segment;

[0206] When n equals N, the first hash value of the Nth file segment is obtained by sampling the Nth file segment;

[0207] When n is less than N and greater than 1, the remainder is obtained by taking the modulo of the first preset value of n, and the remainder is used to sample from one or more sub-segments of the nth file segment to obtain the first hash value of the nth file segment.

[0208] In one embodiment, the first determining module 120 is specifically configured to: when the remainder is 0, divide the nth file segment into sub-segments of a second preset value, and sample the sub-segments using a preset hash algorithm to obtain the first hash value of the nth file segment; when the remainder is 1, extract the first S fields of the nth file segment as the first hash value of the nth file segment using a preset hash algorithm; where S is any positive integer less than the second preset value; when the remainder is greater than or equal to 2, divide the nth file segment into sub-segments corresponding to the remainder of the nth file segment using a preset hash algorithm, and extract one or more bytes from each sub-segment to obtain the first hash value of the nth file segment.

[0209] In one embodiment, the first determining module 120 is specifically used to determine the first hash value of each file segment using a first thread when uploading the file segments;

[0210] The device further includes:

[0211] A first generation module is configured to use a second thread to determine a third hash value of the target file, wherein the third hash value is used to correct the second hash value; wherein the second thread is different from the first thread;

[0212] The sending module 130 is further configured to send the third hash value to the second device.

[0213] like Figure 7 As shown, this disclosure provides a document processing apparatus, the apparatus comprising:

[0214] The receiving module 210 is used to receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided;

[0215] The second determining module 220 is used to determine whether the nth file fragment has been uploaded based on the first hash value;

[0216] The receiving cache module 230 is used to receive and cache the nth file segment uploaded by the first device when the nth file segment has not been uploaded.

[0217] In some embodiments, the file processing apparatus may be a second device.

[0218] In one embodiment, the receiving module 210 may be a network interface or a transceiver antenna. The second determining module 220 and the receiving buffer module 230 may be processors, etc.

[0219] In one embodiment, the receiving module 210, the second determining module 220, and the receiving buffer module 230 may correspond to a processor. The transmitting module may correspond to a network interface or a transceiver antenna, etc. The processor may be a central processing unit, a microprocessor, a digital signal processor, or an embedded processor, etc.

[0220] In another embodiment, the receiving module 210, the second determining module 220, and the receiving buffer module 230 may be program modules; the program modules, after being executed by the processor, can perform the above operations.

[0221] In some embodiments, the receiving module 210, the second determining module 220, and the receiving buffer module 230 may be hardware-software hybrid modules; the hardware-software hybrid modules include, but are not limited to, programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0222] In some embodiments, the receiving module 210, the second determining module 220, and the receiving buffer module 230 may be pure hardware modules; the pure hardware modules include, but are not limited to, various application-specific integrated circuits.

[0223] In some embodiments, the apparatus further includes:

[0224] The second generation module is used to determine the second hash value of the target file based on the first hash values ​​of the first to N file fragments after completing the N first hash values;

[0225] The third determining module is used to determine, based on the second hash value, whether to delete the cached target file or to save the cached target file.

[0226] In some embodiments, the third determining module is specifically configured to perform at least one of the following: determining to save the target file when no fourth hash value identical to the second hash value is found; and determining to delete the cached target file when the fourth hash value identical to the second hash value is found.

[0227] In some embodiments, the receiving module 210 is further configured to receive a third hash value sent by the first device;

[0228] The device further includes:

[0229] The fourth determining module is used to determine whether the second hash value and the third hash value are the same;

[0230] The third determining module is specifically used to determine, based on the second hash value, whether to delete the cached target file or save the cached target file when the second hash value and the third hash value are the same.

[0231] like Figure 8 As shown, this disclosure provides an electronic device, the electronic device comprising:

[0232] Memory;

[0233] The processor, connected to the memory, is configured to implement the file processing method provided in any of the foregoing embodiments by executing computer-executable instructions stored in the memory, such as performing... Figures 1 to 5 Any of the file processing methods shown.

[0234] The electronic device may be the aforementioned first device and / or second device.

[0235] like Figure 8 As shown, the electronic device may also include a network interface, which can be used for information exchange between the first device and the second device.

[0236] This disclosure provides a computer storage medium storing computer-executable instructions; when executed by a processor, these computer-executable instructions can implement the file processing methods provided in any of the foregoing embodiments, for example, performing... Figures 1 to 5 Any file processing method shown. The computer storage medium is a non-transitory storage medium.

[0237] The technical solutions described in the embodiments of this disclosure can be combined arbitrarily without conflict.

[0238] In the several embodiments provided in this disclosure, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0239] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, each functional unit in the various embodiments of this disclosure can be integrated into a second processing unit, or each unit can be a separate unit, or N or more units can be integrated into a unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0241] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A file processing method, characterized in that, Performed by the first device, including: The target file to be transmitted is divided into N file fragments; where N is a positive integer greater than or equal to 2. Determining the first hash value of each of the file segments includes: determining sampling parameters for generating the first hash value of the nth file segment based on the position of the nth file segment among the N file segments; wherein, n is a positive integer less than or equal to N; and sampling the nth file segment according to the sampling parameters to obtain the first hash value of the nth file segment. Based on the first hash value, the N file fragments are sent to the second device; wherein, the first hash value of the N file fragments is used to combine to form the second hash value of the target file; The step of sampling the nth file segment to obtain the first hash value of the nth file segment according to the sampling parameters includes at least one of the following: when n equals 1, sampling the first file segment to obtain the first hash value of the first file segment; when n equals N, sampling the Nth file segment to obtain the first hash value of the Nth file segment; when n is less than N and greater than 1, taking the modulo of n with a first preset value to obtain the remainder, and sampling from one or more sub-segments of the nth file segment according to the remainder to obtain the first hash value of the nth file segment.

2. The method according to claim 1, characterized in that, When n is less than N and greater than 1, taking the modulo of n with a first preset value to obtain a remainder, and sampling from one or more sub-segments of the nth file segment based on the remainder to obtain the first hash value of the nth file segment, includes: When the remainder is 0, the nth file segment is divided into sub-segments of the second preset value, and the sub-segments are sampled using a preset hash algorithm to obtain the first hash value of the nth file segment; When the remainder is 1, the first S fields of the nth file segment are extracted using a preset hash algorithm as the first hash value of the nth file segment; where S is any positive integer less than the second preset value; When the remainder is greater than or equal to 2, the nth file segment is divided into remainder sub-segments corresponding to the nth file segment using a preset hash algorithm, and one or more bytes are extracted from each sub-segment to obtain the first hash value of the nth file segment.

3. The method according to any one of claims 1 to 2, characterized in that, Determining the first hash value of each file fragment includes: using a first thread to determine the first hash value of each file fragment when uploading the file fragment; The method further includes: using a second thread to determine a third hash value of the target file, wherein the third hash value is used to correct the second hash value; wherein the second thread is different from the first thread; The third hash value is sent to the second device.

4. A file processing method, characterized in that, Performed by a second device, the method includes: Receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided; Determine whether the nth file fragment has been uploaded based on the first hash value; When the nth file segment has not been uploaded, the nth file segment uploaded by the first device is received and cached; The method further includes: After completing N first hash values, the second hash value of the target file is determined based on the first hash values ​​of the 1st to Nth file fragments; Based on the second hash value, determine whether to delete the cached target file or save the cached target file; Receive a third hash value sent by the first device; wherein the third hash value is generated by a second thread within the first device, independent of the first thread; the first thread is the thread within the first device that sends the file fragments and generates the first hash value; Determine whether the second hash value and the third hash value are the same; The step of determining whether to delete or save the cached target file based on the second hash value includes: when the second hash value and the third hash value are the same, determining whether to delete or save the cached target file based on the second hash value.

5. The method according to claim 4, characterized in that, The step of determining whether to delete the cached target file or save the cached target file based on the second hash value includes at least one of the following: If no fourth hash value matching the second hash value is found, the target file is saved. When the fourth hash value, which is identical to the second hash value, is retrieved, it is determined that the cached target file will be deleted.

6. A document processing device, characterized in that, include: The file splitting module is used to divide the target file to be transmitted into N file fragments; where N is a positive integer greater than or equal to 2. The first determining module is used to determine the first hash value of each of the file fragments; The sending module is configured to send the N file fragments to the second device according to the first hash value; wherein the first hash value of the N file fragments is used to combine to form the second hash value of the target file; The first determining module is further configured to determine sampling parameters for generating the first hash value of the nth file segment based on the position of the nth file segment among the N file segments; wherein n is a positive integer less than or equal to N; and to sample the nth file segment according to the sampling parameters to obtain the first hash value of the nth file segment; wherein sampling the nth file segment according to the sampling parameters to obtain the first hash value of the nth file segment includes at least one of the following: when n equals 1, sampling the first file segment to obtain the first hash value of the first file segment; when n equals N, sampling the Nth file segment to obtain the first hash value of the Nth file segment; when n is less than N and greater than 1, taking the modulo of n with a first preset value to obtain a remainder, and sampling from one or more sub-segments of the nth file segment according to the remainder to obtain the first hash value of the nth file segment.

7. A document processing device, characterized in that, The device includes: The receiving module is used to receive the first hash value of the nth file fragment uploaded by the first device; wherein, n is a positive integer less than or equal to N; and N is the total number of file fragments into which the target file is divided; The second determining module is used to determine whether the nth file fragment has been uploaded based on the first hash value; A receiving and caching module is configured to receive and cache the nth file segment uploaded by the first device when the nth file segment has not been uploaded. The second determining module is further configured to, after completing N first hash values, determine a second hash value of the target file based on the first hash values ​​of the 1st to Nth file fragments; further configured to, based on the second hash value, determine whether to delete the cached target file or save the cached target file; further configured to receive a third hash value sent by the first device; wherein the third hash value is generated by a second thread within the first device, independent of the first thread; the first thread is the thread within the first device that sends the file fragments and generates the first hash value; the second determining module is further configured to determine whether the second hash value and the third hash value are the same; when the second hash value and the third hash value are the same, the second determining module is further configured to, based on the second hash value, determine whether to delete the cached target file or save the cached target file.

8. An electronic device, characterized in that, The electronic device includes: Memory; A processor, connected to the memory, is configured to implement the file processing method provided by any one of claims 1 to 3 or 4 to 5 by executing computer-executable instructions stored on the memory.

9. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; when the computer-executable instructions are executed by the processor, they can implement the file processing method provided by any one of claims 1 to 3 or 4 to 5.

Images