File uploading method and device, electronic equipment and storage medium

Abstract

The invention discloses a file uploading method and device, electronic equipment and a storage medium. The file uploading method comprises the steps that a client obtains fragments of a to-be-uploaded target file and fragment metadata corresponding to each fragment; sending the meta-information of the target file to a server; the server side judges whether a fragment list corresponding to the meta-information exists or not, if not, the client side generates a fingerprint value of the target file, packages the fingerprint value in fragment metadata corresponding to fragments, and transmits the fragments of the target file and the corresponding fragment metadata to the server side; if yes, the client transmits the fragments of the target file and the corresponding fragment metadata to the server; and for each fragment, the server checks whether the current fragment is stored or not, if not, the current fragment and the corresponding fragment metadata are received and stored, and if the current fragment and the corresponding fragment metadata are stored, skipping is performed. The fragment list is verified before the file is uploaded, so that file verification can be reduced; and the server only receives the fragments which are not stored, so that repeated data receiving can be avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a file uploading method and device, electronic equipment and storage medium. BACKGROUND

[0002] Current breakpoint continuation technology is mostly based on file integrity verification. When a client uploads a large file to a server, if network connection is unstable or the user needs to pause transmission, the entire file transmission can only be abandoned. When the file is re-uploaded, the entire file needs to be re-verified, and the file transmission efficiency is low. SUMMARY

[0003] The present application provides a file uploading method to solve the problem of low file transmission efficiency.

[0004] In a first aspect, the present application provides a file uploading method, comprising:

[0005] The client obtains the shards of a target file to be uploaded and the shard metadata corresponding to each shard.

[0006] The client sends the metadata information of the target file to the server, wherein the metadata information at least includes the file name.

[0007] The server judges whether the shard list corresponding to the metadata information exists, wherein the shard list is used to record the target file, and the shard list corresponds to the metadata information one by one.

[0008] If the shard list corresponding to the metadata information does not exist, the client generates a fingerprint value of the target file, packs the fingerprint value in the shard metadata corresponding to the shard, and transmits the shards of the target file and the corresponding shard metadata to the server.

[0009] If the shard list corresponding to the metadata information exists, the client transmits the shards of the target file and the corresponding shard metadata to the server.

[0010] For each shard, the server checks whether the current shard has been stored. If not, the current shard and the corresponding shard metadata are received and stored. If yes, the current shard and the corresponding shard metadata are skipped.

[0011] In a second aspect, the present application provides a file uploading device, comprising:

[0012] The client is configured to obtain the shards of a target file to be uploaded and the shard metadata corresponding to each shard, and send the metadata information of the target file to the server, wherein the metadata information at least includes the file name.

[0013] The server is configured to determine whether a shard list corresponding to the meta information exists, wherein the shard list is used to record the target file, and the shard list and the meta information correspond to each other in a one-to-one manner.

[0014] The client is further configured to, if the shard list corresponding to the meta information does not exist, generate a fingerprint value of the target file, pack the fingerprint value in shard metadata corresponding to the shard, and transmit the shard of the target file and the corresponding shard metadata to the server; and if the shard list corresponding to the meta information exists, transmit the shard of the target file and the corresponding shard metadata to the server.

[0015] The server is further configured to, for each shard, check whether the current shard has been stored, and if not, receive and store the current shard and the corresponding shard metadata, and if yes, skip the current shard and the corresponding shard metadata.

[0016] In a third aspect, the present application provides an electronic device, which comprises:

[0017] at least one processor; and

[0018] a memory connected to the at least one processor in communication; wherein

[0019] The memory stores a computer program which can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the file uploading method of the first aspect of the present application.

[0020] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for enabling a processor to execute the file uploading method of the first aspect of the present application.

[0021] The file uploading method provided by the embodiments of the present application has the following beneficial effects:

[0022] Before the client sends the file, it is checked whether the shard list corresponding to the meta information of the target file already exists on the server, and if yes, it indicates that the fingerprint value of the target file has been generated before, and thus it is not necessary to generate the fingerprint value by checking the entire target file again, which can improve the efficiency of data uploading; secondly, for all shards of the target file sent by the client, the server only accepts the shards which have not been stored, which can avoid repeated reception of data and improve the speed and efficiency of data uploading.

[0023] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.

[0025] Figure 1 is a flowchart of a file uploading method provided by an embodiment of the present application;

[0026] Figure 2 is a service end processing flowchart provided by an embodiment of the present application;

[0027] Figure 3 is a structural schematic diagram of a file uploading device provided by an embodiment of the present application;

[0028] Figure 4 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the person skilled in the art better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative effort should be within the scope of protection of the present application.

[0030] Figure 1 is a flowchart of a file uploading method provided by an embodiment of the present application. The present embodiment can be applied to the case of file uploading. The method can be executed by a file uploading device. The file uploading device can be realized in the form of hardware and / or software. The file uploading device can be configured in an electronic device. As shown in the figure, the file uploading method comprises the following steps. Figure 1

[0031] In S101, the client acquires the slices of the target file to be uploaded and the slice metadata corresponding to each slice.

[0032] The slices of the target file are files obtained by uniformly cutting the target file. The slice metadata at least includes slice serial number, slice size and offset.

[0033] ​Chunk number is used to define the logical order of the chunk in the complete file. Chunk number can be a continuous integer starting from 0 or 1, which tells the server how to assemble the file in the correct order. For example, "chunkNumber: 3" means the current chunk is the 3rd in order, and "chunkNumber: 5" means the current chunk is the 5th in order.

[0034] Chunk size is used to record the data length of the current chunk, which is used to verify the integrity of the chunk data, especially the last chunk may be smaller than the standard chunk size.

[0035] Chunk offset is used to define the byte-level starting position of the chunk in the original file. This is the key verification information for file recombination. By combining with the total size of the file, it can be verified whether the chunk is in the correct position. For example, "chunkOffset": 10485760, which means the chunk starts from the 10MB of the file.

[0036] In addition, the chunk metadata also includes the fingerprint value of the target file, which is used to associate all chunks to the same parent file. The fingerprint value is generated by the client before uploading starts, ensuring global uniqueness. For example, the fingerprint value is "chunkSize: 5242880".

[0037] Optionally, the client obtains the chunks of the target file to be uploaded and the chunk metadata corresponding to each chunk, including:

[0038] The client determines the chunk size corresponding to the target file to be uploaded according to the network bandwidth condition of the running environment; cuts the target file into multiple chunks according to the chunk size, and obtains the chunk metadata corresponding to each chunk.

[0039] The purpose of this embodiment is to perform a quick network speed test before file uploading, and then select a chunk size that achieves the best balance between efficiency and fault tolerance according to the test result. First, the network delay can be measured by ping or measuring the round-trip time of HTTP requests. The stability of the network can also be judged, and multiple micro uploads are performed within a short period of time to observe the fluctuation of the speed. If the fluctuation is large, it means that the network is unstable.

[0040] The chunk size is related to the stability, reliability and performance in the chunk transmission process. For example, if a large file (such as 2GB) is uploaded as a whole, the connection is interrupted when uploading to 90% in the case of unstable network, and the entire file needs to be re-uploaded, which is very costly. If the file is divided into small chunks of 10MB, if a chunk fails to upload, only the 10MB chunk needs to be re-uploaded, without the need to retransmit the entire 2GB file. This greatly improves the convenience and user experience of uploading.

[0041] S102, the client sends the meta information of the target file to the server, the meta information at least including a file name.

[0042] The meta information further includes a file size and a number of fragments of the target file.

[0043] S103, the server judges whether a fragment list corresponding to the meta information exists.

[0044] The fragment list is used to record a fingerprint value of the target file and fragment metadata and an upload state corresponding to the fragments, and the fragment list corresponds to the meta information one by one.

[0045] The fragment list is a list created when the server first receives the meta information of the target file, and the fragment list is used to record the fragment metadata and the upload state corresponding to the fragments, and the upload state includes uploaded and unuploaded. The fragment list further records the file name of the target file or the fingerprint value generated by the meta information of the target file, so as to correspond to the target file one by one.

[0046] S104, if the fragment list corresponding to the meta information does not exist, the client generates a fingerprint value of the target file, packs the fingerprint value in the fragment metadata corresponding to the fragments, and transmits the fragments of the target file and the corresponding fragment metadata to the server.

[0047] If the fragment list corresponding to the meta information does not exist, it means that the corresponding fragment list also does not exist, and the fingerprint value of the target file has been generated (by the client), at this time, the client generates the fingerprint value of the target file, packs the fingerprint value in the fragment metadata corresponding to the fragments, and transmits the fragments of the target file and the corresponding fragment metadata to the server.

[0048] The fingerprint value can be an MD5 value, and the MD5 value is generated when the entire content of the target file is checked. Illustratively, when calculating the MD5 value of a file, a hash function reads the entire file and regards the file as a long binary data stream. The data stream is divided into blocks of 512 bits, and each block is processed one by one. A series of logical and arithmetic operations are applied to each data block, and the output will affect the calculation of the next data block. After processing the last byte of the file, a unique, fixed-length (128 bits, i.e. 32 hexadecimal characters) hash value is output. The MD5 value generation check is mainly used to verify the integrity and consistency of the file, and by comparing whether the MD5 value generated by the file content is consistent with the original value, it is ensured that the data is not tampered with during transmission or storage.

[0049] Specifically, the client sends the fragments to the server in the form of binary data.

[0050] S105, if the shard list corresponding to the meta information already exists, the client transmits the shards of the target file and the corresponding shard metadata to the server;

[0051] If the shard list corresponding to the meta information already exists, it means that the fingerprint value of the target file already exists. It means that the fingerprint value of the target file has been generated and exists, and the client does not need to regenerate the fingerprint value of the target file. As described before, when generating the fingerprint value, the fingerprint value is packaged in the shard metadata corresponding to the shard. Therefore, in the case where the shard list corresponding to the meta information already exists, the fingerprint value is already packaged in the shard metadata of the shard.

[0052] By detecting the shard list through the embodiment, in the scenario of re-uploading or breakpoint resuming of the target file, the client has already generated the fingerprint value of the target file. At this time, it is not necessary to generate the fingerprint value by re-checking the target file as a whole, which can save the checking time and computing power and improve the data processing and uploading efficiency.

[0053] S106, for each shard, the server checks whether the current shard is already stored, if not, receives and stores the current shard and the corresponding shard metadata, if yes, skips the current shard and the corresponding shard metadata.

[0054] Among them, the shard metadata received by the server is stored in the shard list, and the shard is stored in the preset distributed storage. By retaining only the un-stored shards and the corresponding shard metadata, the same shard is stored only once, saving storage space and improving storage efficiency. It is especially suitable for the scenario of shard uploading and breakpoint resuming.

[0055] Shard uploading is to divide the file to be uploaded into multiple data blocks according to a certain size for separate uploading, and then integrate all the uploaded files into the original file by the server. When the network environment is poor, there is a risk of retransmission.

[0056] Breakpoint resuming is to artificially divide the download or upload task (a file or a compressed package) into several parts when downloading or uploading, and each part is uploaded or downloaded by a thread. If network failure occurs, the unfinished part can be continued to upload or download from the part that has been uploaded or downloaded, without the need to upload or download from the beginning.

[0057] The file uploading method has the beneficial effects that before the client sends a file, it is checked whether the service end has a shard list corresponding to the meta information of the target file, if yes, it indicates that the fingerprint value of the target file has been generated before, and the fingerprint value of the whole target file does not need to be checked again, so that the data uploading efficiency can be improved; secondly, for all shards of the target file sent by the client, the service end only accepts the shards that have not been stored, so that the repeated data receiving can be avoided, and the data uploading speed and efficiency can be improved.

[0058] Optionally, after the service end judges whether the shard list corresponding to the meta information exists, the method further includes: if the shard list corresponding to the meta information does not exist, generating and sending an uploading session ID corresponding to the meta information to the client; and the client starts to perform a data uploading operation after receiving the uploading session ID.

[0059] The service end triggers the client to perform data uploading through the uploading session ID, and the data described herein is the shard of the target file and the shard metadata corresponding to each shard. In some existing front-end controls, for example, a WebUploader control, the control can realize shard cutting on the browser side, supports pause / cancel / continue uploading functions, and the main disadvantage is that it depends on the browser cache, and the file state will be lost after page refreshing; the file shard state management is completely responsible by the client, and the reliability is poor. The embodiment can improve the reliability by managing the file shard state through the service end.

[0060] Optionally, the file uploading method further includes:

[0061] The client sends the shard checking request in multiple threads in parallel;

[0062] The service end checks whether the current shard belongs to the uploading session ID when receiving the shard of the target file, and returns the checking result to the client.

[0063] The real-time checking is performed in the transmission process, instead of the overall checking after all the shards are uploaded, the checking of the data attribution is performed in advance, so that the problem can be found and solved as early as possible. The error shard is prevented from being uploaded to the error session, and the data confusion is avoided. If the session ID is wrong, the client can know it immediately, instead of waiting until the whole transmission is completed, so that the time and bandwidth are saved. In addition, the multiple threads in parallel uploading fully utilizes the network bandwidth, and greatly accelerates the uploading speed.

[0064] Optionally, the file uploading method further includes:

[0065] The server checks the integrity of the current shard according to the shard size in the shard metadata corresponding to the current shard to screen out damaged shards, and when there is a damaged shard, the upload session ID is sent to the client to trigger the client to resend the shards of the target file and the corresponding shard metadata.

[0066] Fast fault detection: immediately find damage at the shard level instead of waiting until the entire file transmission is complete, strong fault tolerance, automatic retry mechanism handles network fluctuations and real-time feedback of data damage; the client can display the upload progress and retransmission status in real time. This embodiment is suitable for transmitting important files in unreliable network environments and can effectively deal with network jitter, packet damage and other transmission problems.

[0067] Optionally, the shard metadata includes a shard serial number, a shard offset and a fingerprint value; the file uploading method further includes:

[0068] The server checks the integrity of the shards under the upload session ID, and if the integrity check fails, the shards under the upload session ID are deleted, and a merge failure information is generated and synchronized to the client; if the integrity check passes, the shard metadata under the upload session ID is sorted and merged to obtain a merged file.

[0069] This stage is the finishing work of the upload process, ensuring the integrity and correctness of the final file. After receiving the "complete upload" instruction from the client, the server will start the verification and construction process.

[0070] The server finds and loads all shard metadata lists corresponding to the session, i.e., shard lists, from storage (such as a database or cache) according to upload_session_id.

[0071] The server will check the integrity of the shard set one by one according to the following rules:

[0072] a. Shards quantity check:

[0073] Rule: The actual number of stored shards must be exactly the same as the total_chunks number declared in the metadata.

[0074] Checking method: Compare the index set of the uploaded shards (such as [0, 1, 2, 3,...]) to see if it is a continuous sequence from 0 to total_chunks - 1 without missing.

[0075] Failure case: If a shard with a missing index is found (for example, there are shards 0, 1, 3, 4, but shard 2 is missing), the check fails.

[0076] b. Shards size check (for the last shard):

[0077] Rule: The actual size of the last chunk (chunk_index = total_chunks - 1) must match the expected size calculated based on the total file size.

[0078] Calculation method: Expected final chunk size = total file size - (standard chunk size * (total_chunks - 1)).

[0079] Failure condition: If the size of the last fragment is not equal to this calculated value, it means that the end of the file may have been truncated or corrupted, and the verification will fail.

[0080] c. Fingerprint value verification (strong consistency guarantee, optional but recommended)

[0081] Rule: The server can recalculate the fingerprint value of each stored fragment and compare it with the fingerprint value provided by the client when uploading.

[0082] Failure scenario: If the fingerprint value calculated by the server for any shard does not match the fingerprint value in the metadata, it indicates that the shard was corrupted before or after storage, and the verification fails.

[0083] Based on the results of the second-stage verification, the process enters two distinct branches.

[0084] Branch A: Validation failed → Cleanup and notification

[0085] Decision: If any of the above verification rules fails, the server determines that there is an unrecoverable data integrity problem in this upload session.

[0086] Perform cleanup:

[0087] The server initiates a cleanup task, deleting all temporary shard files corresponding to the upload_session_id (upload session ID). This is to free up storage space and avoid accumulating useless data.

[0088] At the same time, update the session status to "failed" in the database and record the detailed reasons for failure (e.g., "missing shard #2", "last shard size mismatch").

[0089] Notification to the client:

[0090] The server returns a clear merge_failed response to the client.

[0091] The response should include: a status code (clearly indicating that the merge failed), upload_session_id (indicating which session failed), and an error code and description (so that the client can clearly understand the reason for the failure).

[0092] Branch B: Validation passed → Sorting and merging

[0093] Decision: All integrity checks passed successfully, and the server confirmed that all shards are complete, correct, and intact.

[0094] First, the shards are sorted. The server sorts all shards in ascending order (0, 1, 2, 3, ...) based on the chunk_index in the shard metadata. This order is crucial for correctly reassembling the original files.

[0095] Next, the files are merged. The server creates a new empty file in the specified permanent storage location as the final merged file. Then, in the sorted order, the contents of each fragment file are read out and appended to the final file. This is a sequential I / O operation until the contents of all fragments have been written to the final file in order.

[0096] Optionally, the present invention uses an upload control to implement a series of client functions. The upload control supports file fingerprint generation (MD5 / SHA1 mixed hash), configurable fragment size strategy, fragment metadata record (including fragment sequence number, offset, and fingerprint value), and supports pause, cancel, resume, and retry operations.

[0097] The upload control also supports dynamic configuration of the chunk size. Before the front-end initializes the control, the chunk size configuration item is read as the initial condition for initializing the control, so that the chunk size can be reasonably modified according to the network bandwidth conditions of the running environment.

[0098] Front-end processing flow

[0099] When initializing the control, three core methods need to be registered:

[0100] before-send-file: Generates and verifies the MD5 value before sending the file (whether MD5 verification is needed). Connects to the server to complete the creation and judgment of attachment information initialization. If the server has already archived the attachment information, no additional MD5 value will be generated.

[0101] before-send: Verify before uploading the fragment. This core method will call the server-side fragment verification method. If the fragment has already been uploaded, skip uploading this fragment and the program will automatically verify the next fragment.

[0102] after-send-file: Used to call the server to merge file fragment requests and complete the attachment upload.

[0103] Implement multiple listener methods for personalized front-end rendering;

[0104] The upload control in the client includes the following functions:

[0105] BeforeFileQueued: Triggered before a file is added to the queue, used in conjunction with the BeforeUpload callback method to intercept file uploads;

[0106] FileQueued: Triggered after a file is added to the queue, it starts rendering the list of uploaded files;

[0107] UploadFinished: Triggered after file upload is complete, destroys the upload window; used in conjunction with the afterUpload callback method for custom business logic processing after file upload;

[0108] UploadProgress: Triggered during the upload process, rendering the upload progress bar;

[0109] UploadComplete: Triggered after upload completes, sets the upload window state.

[0110] UploadError: Triggered when an upload file fails, deletes the faulty fragment.

[0111] Figure 2 This is a schematic diagram of a server-side processing flow, such as Figure 2 As shown, the server-side processing flow includes the following steps:

[0112] S201, Initialize upload.

[0113] The server receives file metadata (file name, file size, number of fragments), determines whether the fragment list of the current file has been added to the database, and obtains or generates a unique upload session ID. When a network interruption occurs and the upload is retried, the server will skip the fragment list (fragment data) under the current upload session ID and continue uploading the fragment list (fragment data) that has not yet been uploaded.

[0114] S202, Fragmentation verification.

[0115] The client sends fragment verification requests in parallel using multiple threads. The server identifies whether a fragment has been uploaded, verifies whether the currently uploaded fragment belongs to the current upload session ID, and returns the verification result to the client.

[0116] S203, Segmented storage.

[0117] The client sends file shard storage requests in parallel using multiple threads. The server first performs logical verification on the file shards. If a shard is corrupted, the server automatically clears it and returns the information about the damaged disk, and responds to the client. If the verification passes, the file shards are stored in the distributed storage system. The server records the shard status and returns a success result to the client.

[0118] S204, Fragment merging.

[0119] The server sorts and merges the fragments based on the fragment offsets identified by the current upload session ID. It performs integrity checks on the merged files. If a fragment is lost or corrupted during the merge, the server will fail to merge, delete the fragment identified by the session ID, and send a merge failure error message to the client.

[0120] In this embodiment of the invention, the technical effects achieved based on the file upload method can be summarized into the following four points:

[0121] 1) Improve the reliability of large file transfers: By using a segmented upload and verification mechanism, the entire file transfer is prevented from failing due to network interruption. Only the damaged segments need to be retransmitted, which significantly reduces the transfer failure rate.

[0122] 2) Enhanced transmission efficiency: Supports parallel uploading of fragments, making full use of bandwidth resources and shortening the overall transmission time of large files (especially effective in high-latency networks).

[0123] 3) Flexible user control: The front end provides interactive functions such as pause, cancel, and retry, allowing users to intervene in the transmission process at any time and improve the user experience.

[0124] 4) Data integrity assurance: The server ensures consistency before and after file transmission by performing fragment verification and overall verification after file merging (such as MD5), thus preventing data corruption.

[0125] Figure 3 This is a schematic diagram of a file upload device provided in an embodiment of the present invention. Figure 3 As shown, the file upload device includes a client 31 and a server 32:

[0126] The client is used to obtain the fragments of the target file to be uploaded and the fragment metadata corresponding to each fragment; and to send the metadata of the target file to the server, wherein the metadata includes at least the file name.

[0127] The server determines whether the fragment list corresponding to the metadata already exists. The fragment list is used to record the target file, and the fragment list corresponds one-to-one with the metadata.

[0128] The client is also configured to, if the shard list corresponding to the metadata does not exist, generate a fingerprint value for the target file, package the fingerprint value in the shard metadata corresponding to the shard, and transmit the shards and corresponding shard metadata of the target file to the server; if the shard list corresponding to the metadata already exists, transmit the shards and corresponding shard metadata of the target file to the server.

[0129] On the server side, it is also used to check whether the current shard has been stored for each shard. If not, it receives the current shard and its corresponding shard metadata and stores them. If so, it skips the current shard and its corresponding shard metadata.

[0130] Optionally, the client determines the size of the target file to be uploaded based on the network bandwidth conditions of the operating environment; the target file is then divided into multiple segments according to the segment size, and the segment metadata corresponding to each segment is obtained.

[0131] Optionally, if the shard list corresponding to the metadata does not exist, the server generates an upload session ID corresponding to the metadata and sends it to the client;

[0132] Upon receiving the upload session ID, the client begins the data upload operation.

[0133] Optionally, the client can send fragmented verification requests in parallel using multiple threads;

[0134] When the server receives a fragment of the target file, it verifies whether the current fragment belongs to the upload session ID and returns the verification result to the client.

[0135] Optionally, the server verifies the integrity of the current fragment based on the fragment size in the fragment metadata corresponding to the current fragment to filter out corrupted fragments. When a corrupted fragment exists, the server sends the upload session ID to the client to trigger the client to resend the fragments of the target file and the corresponding fragment metadata.

[0136] Optionally, the fragment metadata includes fragment sequence number, fragment offset, and fingerprint value;

[0137] The server performs integrity checks on the fragments under the upload session ID. If the integrity check fails, the fragments under the upload session ID are deleted, and a merge failure message is generated and synchronized to the client. If the integrity check passes, the server sorts and merges the fragments based on the fragment metadata under the upload session ID to obtain the merged file.

[0138] Optionally, the client is also used to generate a fingerprint value based on the metadata of the target file.

[0139] The file upload device provided in the embodiments of the present invention can execute the file upload method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0140] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0141] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0142] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0143] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as file upload methods.

[0144] In some embodiments, the file upload method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the file upload method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the file upload method by any other suitable means (e.g., by means of firmware).

[0145] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0146] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0147] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0148] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a monitor with a cathode ray tube or liquid crystal display); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0149] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0150] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0151] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0152] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A file uploading method characterized by, The method comprises the following steps: The client obtains the shards of the target file to be uploaded and the shard metadata corresponding to each shard; The client sends the metadata of the target file to the server, wherein the metadata at least includes the file name; The server judges whether the shard list corresponding to the metadata exists, wherein the shard list is used for recording the target file, and the shard list corresponds to the metadata one by one; If the shard list corresponding to the metadata does not exist, the client generates the fingerprint value of the target file, packs the fingerprint value in the shard metadata corresponding to the shard, and transmits the shards of the target file and the corresponding shard metadata to the server; If the shard list corresponding to the metadata already exists, the client transmits the shards of the target file and the corresponding shard metadata to the server; For each shard, the server checks whether the current shard has been stored, if not, the current shard and the corresponding shard metadata are received and stored, if yes, the current shard and the corresponding shard metadata are skipped.

2. The method of claim 1, wherein, The client obtains the shards of the target file to be uploaded and the shard metadata corresponding to each shard, comprising: The client determines the shard size corresponding to the target file to be uploaded according to the network bandwidth condition of the running environment; the target file is cut into multiple shards according to the shard size, and the shard metadata corresponding to each shard is obtained.

3. The method of claim 1, wherein, After the server judges whether the shard list corresponding to the metadata exists, the following steps are further included: If the shard list corresponding to the metadata does not exist, the upload session ID corresponding to the metadata is generated and sent to the client; When the client receives the upload session ID, the data uploading operation is started.

4. The method of claim 3, wherein, Further comprising: The client sends the shard verification request in multiple threads in parallel; When the server receives the shards of the target file, the server checks whether the current shard belongs to the upload session ID, and returns the verification result to the client.

5. The method of claim 3, wherein, Further comprising: The server checks the integrity of the current shard according to the shard size in the shard metadata corresponding to the current shard to filter out the damaged shards, when there are damaged shards, the upload session ID is sent to the client to trigger the client to resend the shards of the target file and the corresponding shard metadata.

6. The method of claim 3, wherein, The shard metadata includes the shard serial number, the shard offset and the fingerprint value; the method further comprises: The server performs integrity check on the shards under the upload session ID, if the integrity check fails, the shards under the upload session ID are deleted, the merging failure information is generated and synchronized to the client; if the integrity check passes, the shard metadata under the upload session ID is sorted and merged to obtain a merged file.

7. The method according to any one of claims 1 to 6, wherein The client generates the fingerprint value of the target file, comprising: The client generates the fingerprint value according to the metadata of the target file.

8. A file uploading apparatus characterized by comprising: The method comprises the following steps: The client obtains the shards of the target file to be uploaded and the shard metadata corresponding to each shard; The client sends the metadata of the target file to the server, wherein the metadata at least includes the file name; The server is configured to determine whether a shard list corresponding to the meta information exists, the shard list being used to record the target file, and the shard list corresponding to the meta information in a one-to-one manner. The client is further configured to, if the shard list corresponding to the meta information does not exist, generate a fingerprint value of the target file, pack the fingerprint value in shard metadata corresponding to a shard, and transmit the shard of the target file and the corresponding shard metadata to the server. If the shard list corresponding to the meta information exists, the shard of the target file and the corresponding shard metadata are transmitted to the server. The server is further configured to, for each shard, check whether the current shard has been stored, and if not, receive and store the current shard and the corresponding shard metadata, and if yes, skip the current shard and the corresponding shard metadata.

9. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the file uploading method in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to enable the processor to execute the file uploading method in any one of claims 1-7 when executed.

Images