Method and apparatus for generating identification information, device, and storage medium

Abstract

A method and apparatus for generating identification information, a device, and a storage medium. The method comprises: obtaining a target file to be sent; then splitting said target file into N file slices; next, respectively encrypting all data in the first file slice and all data in the last file slice to obtain first identification information, and respectively encrypting part of data in each of the second file slice to the (N-1)th file slice to obtain second identification information; and then computing target identification information on the basis of the first identification information and the second identification information. Full sampling is performed on the first file slice and the last file slice (i.e., a computation is performed on all data in each of the file slices to obtain an identifier), partial sampling is performed on the file slices between the first file slice and the last file slice (i.e., a computation is performed on part of data in each of the file slices to obtain an identifier), and finally the target identification information of the entire file to be sent is computed on the basis of the identifiers of all the file slices, so that the speed of identifier generation is improved while ensuring that the identifiers are unique.

Description

Method, device, equipment and storage medium for generating identification information

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202311102340.0 and application date of August 29, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application belongs to the field of information processing technology, and in particular relates to a method, apparatus, device and storage medium for generating identification information. Background Art

[0004] With the development of information technology, users have increasingly higher requirements for the file capacity that can be sent, for example, they need to send large files with a capacity exceeding 10G.

[0005] When sending a file, the user side calculates the data in the file to obtain a unique identifier, and then sends the file and the identifier together to the service receiving end. In the prior art, the conventional message digest algorithm (MD5) is often used to calculate the file identifier. Because the algorithm logic needs to ensure the uniqueness of the identifier, the identifier calculation speed is slow and the generation time is long. For example, on a host (with 32GB of memory and an i7 3.4GB processor), it takes about 20 seconds to calculate the identifier of a 1GB file, and about 17 minutes to calculate the identifier of a 50GB file. It can be seen that the larger the capacity of the sent file, the slower the identifier generation speed. The speed of identifier generation seriously affects the efficiency of file sending and cannot meet user needs.

[0006] Therefore, the existing technology has the problem that it is impossible to ensure the uniqueness of the identifier while improving its calculation speed, thereby affecting the efficiency of file uploading.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a method, apparatus, device, and storage medium for generating identification information, which can improve the calculation speed of an identifier while ensuring its uniqueness, thereby improving the efficiency of file uploading.

[0009] In a first aspect, an embodiment of the present application provides a method for generating identification information, including:

[0010] Get the target file to be sent;

[0011] Split the target file into N fragment files;

[0012] According to a preset encryption algorithm, all data in the target shard file are encrypted to obtain first identification information; partial data in the second to N-1th shard files are encrypted to obtain second identification information; wherein the target shard file includes the first shard file and the last shard file;

[0013] Target identification information is calculated based on the first identification information and the second identification information.

[0014] In some possible implementations, before splitting the target file to obtain N fragment files, the method further includes:

[0015] Determine the capacity threshold of a single shard file based on the server's receiving capacity and network status;

[0016] N is determined based on the total capacity of the target file and the capacity threshold.

[0017] In some possible implementations, determining N based on the total capacity of the target file and a capacity threshold includes:

[0018] When the capacity threshold is greater than or equal to the total capacity of the target file, N is determined to be 1;

[0019] When the capacity threshold is less than the total capacity of the target file, determining a ratio of the capacity of the target file divided by the capacity threshold as a target ratio;

[0020] The target ratio is rounded up to obtain a target integer, and the target integer is used as N.

[0021] In some possible implementations, all data in the target shard file is encrypted and calculated according to a preset encryption algorithm to obtain first identification information; partial data in the second to N-1th shard files is encrypted and calculated to obtain N-2 second identification information, including:

[0022] According to a preset encryption algorithm, encryption calculations are performed on all data in the target fragment file and partial data in the second to N-1th fragment files to obtain first identification information and N-2 second identification information.

[0023] In some possible implementations, the method further includes:

[0024] The capacity ratio of some data in the second shard file is inversely correlated with N.

[0025] In some possible implementations, the method further includes:

[0026] Receive sending status information of the target file and first index information of each sent fragment file, wherein the sending status information and the first index information are obtained by the server based on the target identification information;

[0027] When the sending status information indicates that part of the file has been sent, the sent file is filtered out from the N fragment files to obtain the target fragment file to be sent, wherein the sent fragment file is obtained by querying the preset correspondence between the index information and the fragment file;

[0028] Send the target fragment file and the second index information and identification information of the target fragment file.

[0029] In some possible implementations, sending the target segment file and the second index information and identification information of the target segment file includes:

[0030] The target segment file and the second index information and identification information of the target segment file are sent through a second preset number of network channels.

[0031] In a second aspect, an embodiment of the present application further provides a device for generating identification information, including:

[0032] An acquisition module configured to acquire a target file to be sent;

[0033] A splitting module is configured to split the target file into N fragment files;

[0034] A calculation module is configured to perform encryption calculation on all data in the target shard file according to a preset encryption algorithm to obtain first identification information; and perform encryption calculation on portions of data in the second to N-1th shard files respectively to obtain second identification information; wherein the target shard file includes the first shard file and the last shard file;

[0035] The calculation module is further configured to calculate the target identification information according to the first identification information and the second identification information.

[0036] In some possible implementations, before the segmentation module is configured to segment the target file to obtain N fragment files, the apparatus further includes:

[0037] A determination module configured to determine a capacity threshold of a single shard file based on a server receiving capability and a network status;

[0038] The determination module is further configured to determine N according to the total capacity of the target file and a capacity threshold.

[0039] In some possible implementations, the determination module is further configured to determine N based on the total capacity of the target file and a capacity threshold, including:

[0040] a determining unit configured to determine that N is 1 if the capacity threshold is greater than or equal to the total capacity of the target file;

[0041] The determining unit is further configured to determine, when the capacity threshold is less than the total capacity of the target file, a ratio of the capacity of the target file divided by the capacity threshold as a target ratio;

[0042] The rounding unit is configured to round up the target ratio to obtain a target integer, and use the target integer as N.

[0043] In some possible implementations, the calculation module is configured to perform encryption calculations on all data in the target shard file according to a preset encryption algorithm to obtain first identification information; and perform encryption calculations on portions of data in the second to N-1th shard files to obtain N-2 second identification information, including:

[0044] The computing unit is configured to perform encryption calculation on all data in the target shard file and part of the data in the second to N-1th shard files according to a preset encryption algorithm to obtain first identification information and N-2 second identification information.

[0045] In some possible implementations, the apparatus further includes:

[0046] The capacity ratio of some data in the second shard file is inversely correlated with N.

[0047] In some possible implementations, the apparatus further includes:

[0048] A receiving module configured to receive sending status information of the target file and first index information of each sent fragment file, wherein the sending status information and the first index information are obtained by the server based on the target identification information;

[0049] a filtering module configured to, when the sending status information indicates that the part has been sent, filter out the sent fragment files from the N fragment files to obtain the target fragment files to be sent, wherein the sent fragment files are obtained by querying based on a preset correspondence between the index information and the fragment files;

[0050] The sending module is configured to send the target fragment file and the second index information and identification information of the target fragment file.

[0051] In some possible implementations, the sending module is configured to send the target segment file and the second index information and identification information of the target segment file, including:

[0052] The sending unit is configured to send the target segment file and the second index information and identification information of the target segment file through a second preset number of network channels.

[0053] In a third aspect, an embodiment of the present application further provides a device comprising a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for generating identification information in the first aspect, or any possible implementation of the first aspect, is implemented.

[0054] In a fourth aspect, the present application also provides a computer storage medium, which is computer readable.

[0055] The storage medium stores computer program instructions, which, when executed by a processor, implement the method for generating identification information in the first aspect or any possible implementation of the first aspect.

[0056] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute the method for generating identification information in the first aspect, or any possible implementation of the first aspect.

[0057] The method, apparatus, device and storage medium for generating identification information of the embodiment of the present application obtains the target file to be sent, and then divides the target file into N fragment files. Then, according to the preset encryption algorithm, all the data in the target fragment files are encrypted and calculated to obtain the first identification information, and part of the data in the second to N-1th fragment files are encrypted and calculated to obtain the second identification information, wherein the target fragment files include the first fragment file and the last fragment file. Then, according to the first identification information and the second identification information, the target identification information is calculated. By fully sampling the first fragment file and the last fragment file (i.e., calculating all the data of the fragment file to obtain the identifier), partially sampling the middle fragment file (i.e., calculating part of the data of the fragment file to obtain the identifier), and finally calculating the target identification information of the entire file to be sent based on the identifiers of all the fragment files, the calculation time of the identification information can be greatly saved, the uniqueness of the identifier can be ensured, and the speed of generating the identifier can be increased, thereby improving the efficiency of uploading files. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] FIG1 is a schematic diagram of a process of uploading a target file from a user end to a server end according to an embodiment of the present application;

[0060] FIG2 is a flow chart of a method for generating target identification information provided by an embodiment of the present application;

[0061] FIG3 is a schematic diagram of the workflow of a single WebAssembly instance provided in an embodiment of the present application;

[0062] FIG4 is a flow chart of a method for generating target identification information provided by an embodiment of the present application;

[0063] FIG5 is a schematic diagram of a method for resuming breakpoint transmission of a fragmented file provided in an embodiment of the present application;

[0064] FIG6 is a schematic diagram of a process of managing the upload status of a fragmented file by a state manager according to an embodiment of the present application;

[0065] FIG7 is a schematic diagram of a process of sending a file identifier from a user end to a server end according to an embodiment of the present application;

[0066] FIG8 is a flow chart of a method for uploading fragmented files provided in an embodiment of the present application;

[0067] FIG9 is a schematic diagram of a device for generating identification information provided in an embodiment of the present application;

[0068] FIG10 is a schematic structural diagram of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0070] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, the elements defined by the phrase "comprising..." do not exclude the presence of other identical elements in the process, method, article, or device comprising the elements.

[0071] When uploading a file, you need to first calculate the identifier of the file to be uploaded. The identifier is equivalent to the file's ID number and is a fixed-digit string that must be unique. In existing technologies, such as the conventional MD5 algorithm, all the data in the file is encrypted when calculating the identifier to generate the identifier. Although this ensures the uniqueness of the identifier, as the size of the uploaded file increases, the data involved in the calculation also increases, increasing the amount of calculation and extending the calculation time. This results in slow identifier generation, seriously affecting the upload efficiency of large files.

[0072] Based on this, the embodiment of the present application provides a method, device, equipment and storage medium for generating identification information, which can obtain the target file to be sent, and then split the target file to obtain N fragment files, and then encrypt and calculate all the data in the target fragment files according to the preset encryption algorithm to obtain the first identification information, and at the same time encrypt and calculate part of the data in the second to N-1th fragment files to obtain the second identification information, wherein the target fragment files include the first fragment file and the last fragment file, and then the target identification information can be calculated based on the first identification information and the second identification information. Thus, the speed of generating the identifier can be increased by reducing the amount of data involved in the calculation. At the same time, since all the data of the first fragment file participates in the calculation, the key information of the file will not be missed, and the uniqueness of the identifier can be guaranteed. Therefore, the uniqueness of the identifier can be guaranteed while increasing the speed of generating the identifier, thereby improving the efficiency of sending the file.

[0073] The following describes the technical solution of the embodiment of the present application by taking the user end uploading a file to the server end as an example and combining it with FIG1 .

[0074] The user end refers to the terminal that initiates file upload, such as WEB, H5 or application software App.

[0075] The network refers to the transmission channel for uploading files, such as the public network or local area network.

[0076] The server side refers to a terminal that has the ability to receive files, which may include file storage capabilities, file receiving capabilities, and regular file processing capabilities.

[0077] When uploading a file, the user selects the target file to be uploaded, and then the target file can be split on the user side into N sequentially arranged fragment files. The fragment files can be uploaded to the server side through the network channel. Next, the server side merges the received fragment files to obtain the target file, thus completing the file upload.

[0078] The following is a detailed description of the method for generating identification information provided in the embodiments of the present application with reference to the accompanying drawings.

[0079] FIG2 is a schematic diagram of a method for generating identification information provided in an embodiment of the present application. As shown in FIG2 , the method may include S110 - S140 .

[0080] S110: Obtain the target file to be sent.

[0081] The target file refers to the file to be uploaded, such as a large file with a capacity exceeding 10G.

[0082] Get the target file to be sent, that is, get the target file to be uploaded.

[0083] S120: Split the target file into N fragment files.

[0084] Specifically, step S120 can be understood as dividing the target file to obtain N fragment files. How to determine the number N of fragment files will be described in detail in the following step S150.

[0085] S130, according to a preset encryption algorithm, encrypt all data in the target shard file to obtain first identification information; encrypt part of the data in the second to N-1th shard files to obtain second identification information; wherein the target shard file includes the first shard file and the last shard file.

[0086] Identification information refers to an identifier with a fixed number of bits. Before uploading a file, the user generates this file identification information, also known as a file identifier, using an encryption algorithm. Once generated, the file identifier can be sent to the server for verification. If the corresponding identification information is found in the server's storage space, the file has been successfully uploaded.

[0087] It should be noted that the file identifier is unique.

[0088] An encryption algorithm refers to an algorithm that can generate a file identifier of a fixed number of bits after encrypting the input data. In the embodiment of the present application, the encryption algorithm can be an MD5 algorithm, an MD4 algorithm, an MD3 algorithm or a hash algorithm, etc., which is not limited in the embodiment of the present application.

[0089] Specifically, step S130 can be understood as, according to the preset encryption algorithm, all the data in the target slice file (the target file includes the first slice file and the last slice file) can be encrypted and calculated respectively to obtain identification information, which can be called first identification information; and partial data in the second to N-1th slice files can be encrypted and calculated respectively to obtain identification information, which can be called second identification information. By fully sampling the first slice file and the last slice file (that is, calculating all the data of the slice file to obtain the identifier), the key data of the file will not be missed, thereby ensuring the uniqueness of the identifier, and partially sampling the intermediate slice file (that is, calculating part of the data of the slice file to obtain the identifier). Taking only part of the data of the intermediate slice file can improve the calculation speed of the identifier, but each slice file will take part of the data, ensuring the comprehensiveness of the sampling, thereby achieving the uniqueness of the identifier while improving the generation speed of the identifier.

[0090] In some embodiments, according to a preset encryption algorithm, all data in the target shard file are encrypted and calculated to obtain first identification information; and partial data in the second to N-1th shard files are encrypted and calculated to obtain N-2 second identification information, including:

[0091] According to a preset encryption algorithm, encryption calculations are performed on all data in the target fragment file and partial data in the second to N-1th fragment files to obtain first identification information and N-2 second identification information.

[0092] Specifically, step S130 may include: performing encryption calculations on all data in the target file fragment and partial data in the second to N-1th file fragments according to a preset encryption algorithm to obtain first identification information and N-2 second identification information. By encrypting multiple file fragments simultaneously, the calculation time of the file identifier can be further accelerated, thereby improving the file transmission efficiency.

[0093] In one embodiment, multiple WebAssembly instances can be started at a certain moment to simultaneously calculate the MD5 values ​​of multiple segmented files. WebAssembly (abbreviated as Wasm) is a small and fast binary programming language. Because web applications operate in a single thread, when the main thread calculates the MD5 value, the page will enter a "suspended state" and the browser will have no extra resources to handle human-computer interaction operations. However, by calculating the MD5 value of the segmented file (i.e., the identifier of the segmented file) based on WebAssembly technology, the calculation can be completed with almost no main thread occupied. The workflow of a single WebAssembly is shown in Figure 3. The MD5 algorithm written in C language (i.e., MD5 algorithm-C) is encapsulated and compiled, a WebAssembly instance is generated offline, and then the WebAssembly instance is started to perform parallel calculation of the MD5_k value on the sampled data of a single segmented file (i.e., the entire data of the first segmented file or the last segmented file, or partial data of the segmented file in the middle). This achieves the simultaneous calculation of multiple segmented files and speeds up the calculation of file identifiers overall.

[0094] In some embodiments, the method for generating identification information further includes:

[0095] The capacity ratio of some data in the second shard file is inversely correlated with N.

[0096] Specifically, as the capacity of the target file to be uploaded increases, the more sampling blocks are obtained by splitting (that is, the more fragmented files there are), and the smaller the partial data obtained by sampling the middle fragmented file, that is, the capacity proportion of the partial data in the second fragmented file is inversely correlated with N. This ensures that when the capacity of the file to be uploaded increases, it will not affect the calculation speed of the file identifier. When uploading very large files, the generation time of the file identifier can be controlled within 10s.

[0097] S140: Calculate target identification information based on the first identification information and the second identification information.

[0098] The first identification information is obtained based on full sampling of the first fragment file and the last fragment file, and the second identification information is obtained based on partial sampling of each fragment file in the middle. The first identification information and the second identification information have a one-to-one correspondence with each fragment file, so the total number is N.

[0099] Specifically, step S140 can be understood as calculating the target identification information based on the two first identification information and the N-2 second identification information (i.e., the N identification information corresponding to the N fragmented files). The target identification information is the identification information corresponding to the fragmented file to be uploaded. In the embodiment of the present application, the target identification information of the entire file to be sent can be calculated based on the N identifiers of all fragmented files, which can ensure the uniqueness of the identifier while increasing the speed of identifier generation, thereby improving the efficiency of file transmission.

[0100] It should be noted that the first identification information and the second identification information can be encrypted and calculated according to the MD5 algorithm to obtain the target identification information. In the embodiment of the present application, the algorithm for calculating the target identification information can be an encryption algorithm such as the MD5 algorithm, the MD4 algorithm, the MD3 algorithm or the hash algorithm, which is not limited to this in the embodiment of the present application.

[0101] In an embodiment of the present application, by obtaining a target file to be sent, and then dividing the target file to obtain N fragment files, then according to a preset encryption algorithm, all the data in the target fragment files can be encrypted and calculated to obtain the first identification information, and part of the data in the second to N-1th fragment files can be encrypted and calculated to obtain the second identification information, wherein the target fragment files include the first fragment file and the last fragment file, then according to the first identification information and the second identification information, the target identification information can be calculated. By fully sampling the first fragment file and the last fragment file (i.e., calculating all the data of the fragment file to obtain the identifier), partially sampling the middle fragment file (i.e., calculating part of the data of the fragment file to obtain the identifier), and finally calculating the target identification information of the entire file to be sent through the identifiers of all the fragment files, the calculation time of the identification information can be greatly saved. When uploading a very large file, the generation time of the file identifier can be controlled within 10 seconds, ensuring the uniqueness of the identifier while increasing the generation speed of the identifier, thereby improving the upload efficiency of the file.

[0102] In one embodiment, the file identifier is S file Indicates that the file identifier is calculated as follows:

[0103] Among them, f md5It represents the md5 encryption algorithm, which is an irreversible hash value calculation method. By receiving the binary data of the file, a 32-bit hexadecimal string can be generated. In formula (1), after sampling N fragment files, N md5 values ​​can be obtained: {md5_1, md5_2, ..., md5_k, ..., md5_N-1, md5_N}. Then, the N md5 values ​​are encrypted twice by the MD5 algorithm to obtain the 32-bit file identifier S of the target file. file .

[0104] δ k It is a binary block obtained through a sampling extraction algorithm (i.e., all or part of the data of a shard file).

[0105] It should be noted that the file binary stream (i.e., the target file) primarily includes media description metadata and tracks (video track, audio track, etc.). The metadata includes key information such as the file size, dimensions, and version. Therefore, full sampling of the first fragment allows encryption of this key information, ensuring the uniqueness of the file identifier.

[0106] It should also be noted that, since the size of the last fragment file is uncertain (but does not exceed the capacity of other fragment files), full sampling is adopted in the embodiment of the present application.

[0107] By taking full sampling for the first and last fragment files and partial sampling for the middle fragment files, the sampled binary block δ k The sizes are as follows:

[0108] Among them, ds is the starting index position of the kth shard file, de is the end index position, because δ k The size of is inversely correlated with the number of shard files N, so the end position of the index can be obtained by the number of shard files N.

[0109] In some embodiments, as shown in FIG4 , before splitting the target file to obtain N fragment files, the method further includes:

[0110] S150, determining a capacity threshold of a single shard file according to the server receiving capacity and the network status; and determining N according to the total capacity of the target file and the capacity threshold.

[0111] Specifically, the number N of shard files can be determined by the server's receiving capacity and network status. Specifically, the capacity threshold of a single shard file can be determined based on the server's receiving capacity and network status. The number N of shard files can then be determined based on the total capacity of the target file and the capacity threshold. This can be dynamically set based on network quality, reducing the upload failure rate and ensuring efficient file uploads.

[0112] In some embodiments, determining N based on the total capacity of the target file and a capacity threshold includes:

[0113] When the capacity threshold is greater than or equal to the total capacity of the target file, N is determined to be 1;

[0114] When the capacity threshold is less than the total capacity of the target file, determining a ratio of the capacity of the target file divided by the capacity threshold as a target ratio;

[0115] The target ratio is rounded up to obtain a target integer, and the target integer is used as N.

[0116] Specifically, according to the total capacity of the target file and the capacity threshold, N is determined, including: file ≥Total capacity of target file N file In this case, N=1; at the capacity threshold T file <Total capacity of target file N file In this case, determine the target ratio = N file / T file , then round up the target ratio to get the target integer, and use the target integer as N.

[0117] In one embodiment, the total size of the target file to be uploaded is N file (unit: MB), the capacity threshold of a single shard file is T file =4MB, the calculation formula for the number of fragment files N is:

[0118] Among them, math.ceil is the rounding function.

[0119] In some embodiments, as shown in FIG5 , the method for generating identification information further includes:

[0120] S161, receiving sending status information of the target file and first index information of each sent fragment file, wherein the sending status information and the first index information are obtained by the server based on the target identification information;

[0121] S162: If the sending status information indicates that the file has been partially sent, filter out the sent file fragments from the N file fragments to obtain the target file fragments to be sent, wherein the sent file fragments are obtained by querying the preset correspondence between the index information and the file fragments;

[0122] S163: Send the target fragment file and the second index information and identification information of the target fragment file.

[0123] Specifically, step S161 can be understood as, after the server receives the file identifier of the target file to be uploaded, it will query the storage space for the corresponding target file's sending status information based on the file identifier. For example, the sending status information can be "not uploaded," "partially uploaded," or "fully uploaded." If the sending status information indicates "partially uploaded," the server will extract the index information of the uploaded segmented files, which can be referred to as first index information. Therefore, the user end can receive the sending status information of the target file and the first index information of each segmented file that has been sent.

[0124] Specifically, step S162 can be understood as, based on the preset correspondence between the index information and the fragmented files, it can be queried to obtain which fragmented file corresponds to the first index information. Therefore, when the sending status information is partially sent, among the N fragmented files, the sent fragmented files corresponding to the first index information can be filtered out according to the preset correspondence to obtain the target fragmented file to be sent.

[0125] Specifically, step S163 can be understood as the user terminal sending the target segment file to be sent and the index information and identification information of the target segment file. The index information can be called second index information.

[0126] In an embodiment of the present application, by receiving the sending status information of the target file and the first index information of each fragment file that has been sent, wherein the sending status information and the first index information are obtained by the server based on the target identification information query, then when the sending status information is partially sent, in the N fragment files, the sent fragment files obtained by the query based on the preset correspondence between the index information and the fragment files are filtered out to obtain the target fragment file to be sent, and then the target fragment file and the second index information and identification information of the target fragment file can be sent. Through the first index information, it is possible to know which corresponding fragment files have been successfully uploaded, so that only the fragment files that have not been uploaded can be uploaded, realizing the resume of disconnection, avoiding the repeated uploading of the uploaded fragment files, and further improving the efficiency of file uploading. When users transmit the same file in different time periods, and when different users upload the same file on the same or different terminals, it can be determined whether the file has been uploaded through the first index information, and there is no need to upload it from the beginning, thereby avoiding repeated uploading.

[0127] In some embodiments, sending the target segment file and the second index information and identification information of the target segment file includes:

[0128] The target segment file and the second index information and identification information of the target segment file are sent through a second preset number of network channels.

[0129] The number of network channel settings can be called a second preset number, that is, the number of fragmented files uploaded in parallel, which can be dynamically changed according to the performance of the browser. For example, the second preset number is 4.

[0130] Specifically, the number of network channels is also a significant factor influencing file upload efficiency. Each file fragment can be transmitted independently. By transmitting the fragments in parallel through a second preset number of network channels, the order in which the fragments are uploaded no longer depends on the order in which they are uploaded. For example, if the second preset number is 4, the upload speed can be 4 times that of a single channel upload, thereby increasing the upload speed and efficiency of the fragmented files.

[0131] In one embodiment, when uploading segmented files in parallel through multiple network channels, the upload status of the segmented files can be managed, and the status manager of multi-channel network parallel upload is divided into four queues, namely: the http queue to be uploaded, the http queue in progress, the http collection of failed uploads, and the http collection of completed uploads. The status flow process is shown in Figure 6.

[0132] As shown in Figure 6, the state flow process is as follows:

[0133] 1) After the start, the shard file state manager slices the entire target file and filters the shard files that have been uploaded in the past, so as to create a queue of HTTP instances to be uploaded;

[0134] 2) The status manager monitors in real time whether the length of the uploading http queue (i.e., the number of fragmented files being uploaded) is less than the number of parallel uploads that can be received, Nttp. If it is less than Nttp, a new fragmented file http upload instance can be created and added to the uploading http queue.

[0135] It should be noted that browsers usually allow about 6 parallel http connections, so in the embodiment of the present application, Nttp can be 4, which can ensure 1-2 channels for business http use.

[0136] 3) After the HTTP instance of a fragmented file in the uploading HTTP queue completes the transmission, it will be stored in the completed HTTP instance, and then the HTTP queue to be uploaded will be triggered to perform the above 1) and 2) operations;

[0137] 4) When an exception occurs in the uploading HTTP queue, the transmission is terminated, the instance is stored in the failed HTTP collection, and the transmission ends;

[0138] 5) When both the upload queue and the uploading HTTP queue are empty, it means that all the segmented files have been uploaded and the process ends.

[0139] In one embodiment, the client includes a "file slicing module," a "file identifier calculation module," a "file status checking module," and a "sliced ​​file status management module," while the server includes a "file status management module," a "file status synchronization module," a "merging module," and a "verification module." A flowchart for the client sending a file identifier to the server is shown in Figure 7, and a flowchart for uploading a sliced ​​file is shown in Figure 8.

[0140] The flowchart of the user side sending the file identifier to the server side is shown in Figure 7:

[0141] 1) The "file slicing module" on the user side first obtains the target file to be uploaded, calculates the number of slice files N according to the above formula (3), and then segments the target file to obtain N slice files and sends them to the "file identifier calculation module". For example, the "file slicing module" can use the File.slice method to segment the entire file. The capacity of each slice file is Tfile. Since the last slice may be less than Tfile, the file is segmented according to the actual capacity. In this embodiment of the application, file slicing refers to the reference of file blocks, and does not load the entire file and blocks into memory, which can minimize the use of memory resources.

[0142] 2) The "file identifier calculation module" calculates the file identifier according to the above formula (1) and formula (2), and sends the calculated identifier to the "file status check module";

[0143] 3) The "file status check module" sends the identifier to the "file status management module" on the server side;

[0144] 4) The "file status management module" queries the server-side storage space for the file status information of the target file based on the identifier. If the file status information indicates that the file has been partially uploaded, it queries the index information of the uploaded segmented files and returns the file status information and the index information of the uploaded segmented files to the "file status checking module" on the user side.

[0145] It should be noted that the upload status information of the entire file may include: not uploaded, partially uploaded, fully uploaded and fully merged, and fully uploaded but not fully merged.

[0146] 5) The "file status check module" on the user side will receive the file status information and index information of the uploaded segmented files returned by the server side. If the file status information shows that the file has not been uploaded, the N segmented files will be uploaded; if the file status information shows that all the segmented files have been uploaded and all the segmented files have been merged, the process will be successfully completed; if the file status information shows that all the segmented files have been uploaded but not all the segmented files have been merged, the process will be successfully completed after all the segmented files have been merged; if the file status information shows that some of the segmented files have been uploaded, the process will be as shown in Figure 8.

[0147] The flowchart of shard file upload is shown in Figure 8:

[0148] 6) The file status information returned by the "file status management module" on the server side is partially uploaded, and the index information of the uploaded fragmented files is returned to the "file status checking module" on the user side;

[0149] 7) The "file status check module" filters the uploaded fragment files from the N fragment files based on the index information to obtain the target fragment file to be uploaded. The "fragment file status management module" then uploads the target fragment file and its index information and identifier sign to the "verification module" on the server side, thus enabling breakpoint resuming.

[0150] It should be noted that the upload status information of the segmented file may include: not uploaded and uploaded.

[0151] 8) The "verification module" calculates the identifier sign' based on the target shard file and the above formulas (1) and (2). When sign'=sign, it indicates that the received target shard file is correct and has not been mutated during the file transmission process. Next, the target shard file and its index information are stored in the storage space on the server side for the server side to merge the shard files and synchronize the file status information. The verification result is returned to the "shard file status management module" on the user side.

[0152] 9) The "segment file status management module" determines whether there are segment files that have failed to be uploaded based on the verification result. If there are segment files that have failed to be uploaded, the process ends. If there are no segment files that have failed to be uploaded, the process returns to the "file status check module" to receive the file status information updated by the server-side "file status management module";

[0153] 10) After the verification module on the server side successfully verifies, the status information of the successfully uploaded segment files will be updated from not uploaded to uploaded, and the "merge module" will merge the segment files whose status information has changed (i.e., updated from not uploaded to uploaded) with other adjacent successfully uploaded segment files. It is not necessary to start merging after the last segment file is successfully uploaded. In the embodiment of the present application, as long as one segment file is uploaded, the segment file can be merged with the adjacent successfully uploaded segment files, which reduces the user's waiting time and improves the efficiency of file uploading;

[0154] 11) The "merge module" on the server side can realize the simultaneous uploading and merging of fragmented files. The "merge module" sends the merge result to the "file status synchronization module", and the "file status synchronization module" sends the updated file status information to the "file status management module".

[0155] For example, after each fragment file is received, the server will start looking for surrounding continuous fragment files that can be merged. After fragment file_1, fragment file_2, and fragment file_4 are uploaded, the control program recognizes that fragment file_1 and fragment file_2 are continuous fragments and starts merging them into a larger fragment file. Then, when fragment file_5 is uploaded, the control program recognizes that fragment file_4 and fragment file_5 are continuous fragments and starts merging them into a larger fragment file. There is no need to wait for fragment file_3 to be uploaded before starting the merge. As long as adjacent continuous fragment files are detected, the merge can be started. After each fragment file or continuous block is merged, the original fragment file or block file can be deleted to save disk space. When the last fragment file_N is uploaded successfully, the merge operation of fragment file_N and fragment file_1 to fragment file_N-1 is started. In an embodiment of the present application, you only need to wait for the merging time of a fragment file, and there is no need to start merging after the last fragment file is uploaded. The merging time can be 1 / N of the normal merging time, which greatly reduces the user's waiting time and improves the file upload efficiency.

[0156] The embodiment of the present application further provides a device for generating identification information. As shown in FIG9 , the device 900 includes an acquisition module 910 , a segmentation module 920 , and a calculation module 930 :

[0157] An acquisition module 910 is configured to acquire a target file to be sent;

[0158] The splitting module 920 is configured to split the target file into N fragment files;

[0159] The calculation module 930 is configured to perform encryption calculation on all data in the target shard file according to a preset encryption algorithm to obtain first identification information; and perform encryption calculation on partial data in the second to N-1th shard files respectively to obtain second identification information; wherein the target shard file includes the first shard file and the last shard file;

[0160] The calculation module 930 is further configured to calculate the target identification information according to the first identification information and the second identification information.

[0161] In an embodiment of the present application, the device for generating identification information obtains the target file to be sent, and then divides the target file into N fragment files. Next, according to a preset encryption algorithm, all the data in the target fragment files are encrypted and calculated to obtain the first identification information, and part of the data in the second to N-1th fragment files are encrypted and calculated to obtain the second identification information, wherein the target fragment files include the first fragment file and the last fragment file. Next, according to the first identification information and the second identification information, the target identification information is calculated. By fully sampling the first fragment file and the last fragment file (i.e., calculating all the data of the fragment file to obtain the identifier), partially sampling the middle fragment file (i.e., calculating part of the data of the fragment file to obtain the identifier), and finally calculating the target identification information of the entire file to be sent through the identifiers of all the fragment files, the calculation time of the identification information can be greatly saved, the uniqueness of the identifier can be ensured, and the speed of generating the identifier can be increased, thereby improving the efficiency of uploading files.

[0162] In some embodiments, before the segmentation module is configured to segment the target file to obtain N fragment files, the apparatus further includes:

[0163] A determination module configured to determine a capacity threshold of a single shard file based on a server receiving capability and a network status;

[0164] The determination module is further configured to determine N according to the total capacity of the target file and a capacity threshold.

[0165] In some embodiments, the determination module is further configured to determine N based on the total capacity of the target file and a capacity threshold, including:

[0166] a determining unit configured to determine that N is 1 if the capacity threshold is greater than or equal to the total capacity of the target file;

[0167] The determining unit is further configured to determine, when the capacity threshold is less than the total capacity of the target file, a ratio of the capacity of the target file divided by the capacity threshold as a target ratio;

[0168] The rounding unit is configured to round up the target ratio to obtain a target integer, and use the target integer as N.

[0169] In some embodiments, the calculation module is configured to perform encryption calculation on all data in the target shard file according to a preset encryption algorithm to obtain first identification information; and perform encryption calculation on part of the data in the second to N-1th shard files to obtain N-2 second identification information, including:

[0170] The computing unit is configured to perform encryption calculation on all data in the target shard file and part of the data in the second to N-1th shard files according to a preset encryption algorithm to obtain first identification information and N-2 second identification information.

[0171] In some embodiments, the apparatus further comprises:

[0172] The capacity ratio of some data in the second shard file is inversely correlated with N.

[0173] In some embodiments, the apparatus further comprises:

[0174] A receiving module configured to receive sending status information of the target file and first index information of each sent fragment file, wherein the sending status information and the first index information are obtained by the server based on the target identification information;

[0175] a filtering module configured to, when the sending status information indicates that the part has been sent, filter out the sent fragment files from the N fragment files to obtain the target fragment files to be sent, wherein the sent fragment files are obtained by querying based on a preset correspondence between the index information and the fragment files;

[0176] The sending module is configured to send the target fragment file and the second index information and identification information of the target fragment file.

[0177] In some embodiments, the sending module is configured to send the target segment file and the second index information and identification information of the target segment file, including:

[0178] The sending unit is configured to send the target segment file and the second index information and identification information of the target segment file through a second preset number of network channels.

[0179] The various modules in the device for generating identification information provided in the embodiment of the present application can implement the functions of the various steps of the method for generating identification information provided in Figures 1 to 8, and can achieve the corresponding technical effects. For the sake of concise description, they will not be repeated here.

[0180] FIG10 shows a schematic diagram of the hardware structure of a device for generating identification information provided in an embodiment of the present application.

[0181] The device for generating identification information may include a processor 1001 and a memory 1002 storing computer program instructions.

[0182] Specifically, the processor 1001 may include a central processing unit (CPU) or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0183] The memory 1002 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1002 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 1002 may include removable or non-removable (or fixed) media. Where appropriate, the memory 1002 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 1002 is a non-volatile solid-state memory.

[0184] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of the present application.

[0185] The processor 1001 reads and executes computer program instructions stored in the memory 1002 to implement any one of the identification information generation methods in the above embodiments.

[0186] In one example, the device for generating identification information may further include a communication interface 1003 and a bus 1004. As shown in FIG10 , the processor 1001, the memory 1002, and the communication interface 1003 are connected via the bus 1004 and communicate with each other.

[0187] The communication interface 1003 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0188] Bus 1004 includes hardware, software, or both that couples components of the terminal services device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Linear Predictive Coding (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (Peripheral Component Interconnect-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VESA Local Bus) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1004 may include one or more buses. Although embodiments herein describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

[0189] The device can execute the identification information generation method in the embodiment of the present application based on the various units / components in the identification information generation apparatus, thereby realizing the identification information generation method described in combination with Figures 1 to 8.

[0190] In addition, in conjunction with the method for generating identification information in the above embodiments, embodiments of the present application may provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any of the methods for generating identification information in the above embodiments is implemented.

[0191] The present application also provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes each process of any one of the above-mentioned methods for generating identification information.

[0192] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0193] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable read-only memories (EROMs), floppy disks, compact disc read-only memories (CD-ROMs), optical discs, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0194] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0195] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0196] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.

Claims

1. A method for generating identification information, comprising: Get the target file to be sent; Splitting the target file to obtain N fragment files; According to a preset encryption algorithm, all data in the target slice file are encrypted and calculated to obtain first identification information; partial data in the second to N-1th slice files are encrypted and calculated to obtain second identification information; wherein the target slice file includes the first slice file and the last slice file; Target identification information is calculated based on the first identification information and the second identification information.

2. The method according to claim 1, wherein: Before the target file is segmented to obtain N fragment files, the method further includes: Determine the capacity threshold of a single shard file based on the server receiving capacity and network status; N is determined according to the total capacity of the target file and the capacity threshold.

3. The method according to claim 2, wherein: The determining N according to the total capacity of the target file and the capacity threshold comprises: When the capacity threshold is greater than or equal to the total capacity of the target file, determining N to be 1; In a case where the capacity threshold is less than the total capacity of the target file, determining a ratio of the capacity of the target file divided by the capacity threshold as a target ratio; The target ratio is rounded up to obtain a target integer, and the target integer is used as N.

4. The method according to claim 1, wherein: According to the preset encryption algorithm, all the data in the target shard file are encrypted and calculated to obtain the first identification information; and partial data in the second to N-1th shard files are encrypted and calculated to obtain N-2 second identification information, including: According to a preset encryption algorithm, encryption calculation is performed on all data in the target fragment file and partial data in the second to N-1th fragment files to obtain first identification information and N-2 second identification information.

5. The method according to claim 1, wherein: Also includes: The capacity proportion of the partial data in the second shard file is inversely correlated with N.

6. The method according to any one of claims 1 to 5, wherein: Also includes: Receive the sending status information of the target file and the first index information of each segment file that has been sent, wherein the sending status information and the first index information are obtained by the server based on the target identification information query; In the case where the sending status information indicates that part of the fragments have been sent, the sent fragments are filtered out from the N fragments to obtain the target fragments to be sent, wherein the sent fragments are obtained by querying based on a preset correspondence between the index information and the fragments; Send the target segment file and the second index information and identification information of the target segment file.

7. The method according to claim 6, wherein: The sending of the target slice file and the second index information and identification information of the target slice file includes: The target segment file and the second index information and identification information of the target segment file are sent through a second preset number of network channels.

8. A device for generating identification information, comprising: An acquisition module, configured to acquire a target file to be sent; A segmentation module is configured to segment the target file to obtain N segment files; The calculation module is configured to perform encryption calculation on all data in the target shard file according to a preset encryption algorithm to obtain the first identification information; and perform encryption calculation on the second to the N-1th shard files respectively. Perform encryption calculation on part of the data in the slice file to obtain second identification information; wherein the target slice file includes the first slice file and the last slice file; The calculation module is further configured to calculate target identification information according to the first identification information and the second identification information.

9. A device, comprising: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for generating identification information according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the method for generating identification information according to any one of claims 1 to 7.

Images