File transmission method and device, electronic equipment and storage medium

Abstract

Provided are a file transmission method and device, electronic equipment, and storage medium. The file transmission method comprises: receiving streaming data of a target file from a client; performing slice processing on the streaming data to obtain file slice data of the target file; and storing the file slice data of the target file, wherein the streaming data is generated based on first file slice data of the target file obtained by the client performing slice processing on the target file; the streaming data of the target file comprises two or more target data blocks, each target data block comprising a first file slice data and a corresponding slice serial number; and the slice serial number is used to indicate the order of the corresponding first file slice data in the target file relative to other first file slice data.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and specifically to a file transfer method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, most file transfer technologies utilize concurrent chunked uploads. In this technology, a request is sent to the server for each chunk to be transferred. The next chunk is only transferred after the current chunk has finished (e.g., after receiving a response from the file service). The waiting time (i.e., network link idle time) between the completion of chunk transmission and the receipt of the server's response often reaches hundreds or thousands of milliseconds. When the number of chunks is high, this leads to a significant overall idle time.

[0003] Streaming upload technology is an effective way to solve the problem of high network link idle time in chunked upload technology. It eliminates the need for hundreds or thousands of requests to the server, allowing file uploads to be completed with only one or a few network requests, thus reducing idle time. However, traditional streaming upload technology requires the server to use disk space to store large files during transmission. This is because memory space is insufficient for processing large files, easily leading to server memory write amplification problems. Summary of the Invention

[0004] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, according to one or more embodiments of this disclosure, a file transfer method is provided, applied on a server, comprising:

[0006] Receive streaming data from the target file in the client;

[0007] The streaming data is sliced ​​to obtain file fragment data of the target file;

[0008] Store the file fragment data of the target file;

[0009] The streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block contains a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file.

[0010] Secondly, according to one or more embodiments of this disclosure, a file transfer method is provided, applied to a client, comprising:

[0011] The target file to be transmitted is sliced ​​to obtain the first file slice data of the target file;

[0012] Send streaming data containing the first file fragment data to the server so that the server performs the following steps: receive the streaming data; slice the streaming data to obtain the file fragment data of the target file; and store the file fragment data of the target file.

[0013] The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0014] Thirdly, according to one or more embodiments of this disclosure, a file transfer apparatus is provided, comprising:

[0015] The receiving unit is used to receive streaming data from the target file of the client;

[0016] A slicing unit is used to slice the streaming data to obtain file fragment data of the target file;

[0017] Storage unit, used to store file fragment data of the target file;

[0018] The streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block contains a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file.

[0019] Fourthly, according to one or more embodiments of this disclosure, a file transfer apparatus is provided, comprising:

[0020] A slicing unit is used to slice the target file to be transmitted to obtain the first file fragment data of the target file;

[0021] The sending unit is configured to send streaming data containing the first file fragment data to the server, so that the server performs the following steps: receiving the streaming data; performing segmentation processing on the streaming data to obtain the file fragment data of the target file; and storing the file fragment data of the target file.

[0022] The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0023] Fifthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one memory and at least one processor; wherein the memory is configured to store program code, and the processor is configured to invoke the program code stored in the memory to cause the electronic device to perform a file transfer method provided according to one or more embodiments of the present disclosure.

[0024] In a sixth aspect, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, the non-transitory computer storage medium storing program code, which, when executed by a computer device, causes the computer device to perform a file transfer method provided according to one or more embodiments of the present disclosure.

[0025] According to one or more embodiments of this disclosure, by receiving streaming data of a target file from a client, slicing the streaming data to obtain file fragment data of the target file, and storing the file fragment data of the target file, the network link idle time generated during the transmission of the target file can be reduced through streaming transmission. Furthermore, by slicing the received streaming data on the server side, the file fragment data of the target file can be quickly stored and reclaimed from memory, thereby reducing the memory usage when storing the target file on the server side and avoiding the memory write amplification problem. Moreover, by having the client first slice the target file to be transmitted locally, the streaming data of the target file contains multiple target data blocks, each containing the file sequence number of the target file and the corresponding fragment sequence number, thus providing a basis for implementing breakpoint resumption and concurrent out-of-order transmission of file fragments. Attached Figure Description

[0026] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0027] Figure 1 This is a flowchart of a file transfer method provided according to an embodiment of the present disclosure;

[0028] Figure 2 This is a schematic diagram of the structure of a file transfer system according to an embodiment of the present disclosure.

[0029] Figure 3 This is a signal flow diagram of a file transfer system provided according to an embodiment of the present disclosure;

[0030] Figure 4 This is a flowchart of a file transfer method provided according to another embodiment of the present disclosure;

[0031] Figure 5 This is a schematic diagram of the structure of a file transfer device according to an embodiment of the present disclosure;

[0032] Figure 6 This is a schematic diagram of a file transfer device according to another embodiment of the present disclosure;

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0034] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] It should be understood that the steps described in the embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended inclusion, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The term "in response to" and related terms mean that one signal or event is affected to some extent by another signal or event, but not necessarily completely or directly. If event x occurs "in response to" event y, then x may be directly or indirectly responsive to y. For example, the occurrence of y may ultimately lead to the occurrence of x, but there may be other intermediate events and / or conditions. In other cases, y may not necessarily lead to the occurrence of x, and x may occur even if y has not yet occurred. Furthermore, the term "in response to" can also mean "at least partially responsive to".

[0037] The term "determine" broadly encompasses a wide variety of actions, including acquisition, calculation, computation, processing, derivation, investigation, search (e.g., searching in a table, database, or other data structure), discovery, and similar actions; it may also include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), parsing, selecting, choosing, building, and similar actions, etc. Definitions for other terms will be provided below.

[0038] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0039] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0040] For the purposes of this disclosure, the phrase “A and / or B” means (A), (B) or (A and B).

[0041] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0042] refer to Figure 1 , Figure 1 A flowchart of a file transfer method 100 provided in an embodiment of the present disclosure is shown. The method 100 includes steps S120-S160.

[0043] Step S120: Receive streaming data from the target file on the client.

[0044] In this embodiment, the client uses streaming technology to upload the target file to the server. Compared with traditional fragmented upload technology, it can reduce network link idle time. Streaming data typically refers to a sequence of data that arrives sequentially, in large quantities, quickly, and continuously. Generally, streaming data can be considered as a dynamic data set that grows over time.

[0045] Step S140: Slice the streaming data to obtain file fragment data of the target file.

[0046] Step S160: Store the file fragment data of the target file.

[0047] In some embodiments, the server can slice the currently received streaming data in real time based on a preset slice data size, thereby obtaining the file slice data of the target file in real time and storing the obtained file slice data in real time.

[0048] In some embodiments, file fragment data can be stored in memory first, and then the file fragment data in memory can be persisted, for example, stored in a disk or database. The persisted file fragment data is then released from memory, thereby ensuring fast storage of fragment data and real-time memory reclamation, and optimizing the memory usage of large file uploads.

[0049] In one specific implementation, multiple coroutines or threads can be started to slice the streaming data of the target file, verify the accuracy of the sliced ​​data, encrypt and store the sliced ​​data, and record the metadata of the sliced ​​data in the database, but this disclosure is not limited thereto.

[0050] In some embodiments, the streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block containing a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file.

[0051] In this embodiment, the client does not directly stream the target file. Instead, it first performs slicing processing on the target file locally, dividing it into multiple file fragments (i.e., first file fragment data), and then streams these multiple first file fragment data to the client. In one specific implementation, assuming the client divides the target file locally into N first file fragment data, such as fragment 1, fragment 2, fragment 3, ..., fragment N-1 and fragment N arranged in sequence, then the request body of the HTTP (Hypertext Transfer Protocol) request sent by the client to the server can contain N connected target data blocks (or target data segments). Each target data block contains a fragment sequence number (e.g., sequence number 1, 2, 3, ..., N-1 or N) and the first file fragment data corresponding to that fragment sequence number.

[0052] According to one or more embodiments of this disclosure, by receiving streaming data of a target file from a client, slicing the streaming data to obtain file fragment data of the target file, and storing the file fragment data of the target file, the network link idle time generated during the transmission of the target file can be reduced through streaming transmission. Furthermore, by slicing the received streaming data on the server side, the file fragment data of the target file can be quickly stored and reclaimed from memory, thereby reducing the memory usage when storing the target file on the server side and avoiding the memory write amplification problem. Moreover, by having the client first slice the target file to be transmitted locally, the streaming data of the target file contains multiple target data blocks, each containing the file sequence number of the target file and the corresponding fragment sequence number, thus providing a basis for implementing breakpoint resumption and concurrent out-of-order transmission of file fragments.

[0053] Furthermore, in some embodiments, the target data block may also contain other relevant information, such as target verification information corresponding to the first segment file data. The server calculates the target verification information based on the file segment data of the target file obtained through its own slicing process, and compares the calculated target verification information with the target verification information already carried in the target data block. If they are the same, it indicates that the file segment data in the target data block is accurate; if they are different, it indicates that the file segment data in the target data block is inaccurate, for example, it may have been tampered with by a third party during data transmission.

[0054] In one specific implementation, the target verification information can be a CRC32 value, which is a 32-bit cyclic redundancy check code.

[0055] Furthermore, in some embodiments, the streaming data is transmitted concurrently via two or more links, with each link streaming at least one first file fragment. For example, multiple threads can be used to concurrently stream the first file fragments of the target file, with each thread responsible for transmitting one or more first file fragments. In this embodiment, the target file is divided into multiple file fragments on the client side, and each fragment is identified by a fragment number in the streaming data. This allows the server to know the order of each file fragment, enabling multi-threaded streaming of the target file's file fragments without being affected by the disorder of multi-threaded execution. Moreover, traditional streaming technologies often use single long-lived connections for data transmission. This single-connection-based data transmission method is easily affected by network fluctuations, resulting in slow data transmission. This embodiment, however, uses concurrent streaming to ensure that other connections can continue transmitting even when one connection is broken, thus efficiently utilizing the user's upload bandwidth.

[0056] In some embodiments, the file fragment data of the target file includes the first file fragment data. In one specific embodiment, after the server receives the streaming data of the target file, the server performs slicing processing to cut each first file fragment data or the target data block containing the first file fragment data from the continuous streaming data, thereby obtaining the file fragment data obtained by the client through slicing processing. The file fragment data obtained by the server from the streaming data of the target file through slicing processing corresponds one-to-one with the file fragment data obtained by the client from the target file through slicing processing.

[0057] Furthermore, some embodiments of the file transfer method also include:

[0058] Step A1: Based on the fragment sequence number corresponding to the first file fragment data obtained from the server and the total number of first file fragments contained in the target file, determine the first file fragment data that was not successfully transmitted to the server;

[0059] Step A2: Send a preset error message to the client, the error message indicating that the first file fragment data was not successfully transmitted to the server.

[0060] In this embodiment, the server can identify the segment number corresponding to the currently obtained first file segment data by recognizing the segment number within the target data block. This allows the server to ultimately know the segment numbers of all uploaded first file segment data. Combined with the total number of first file segments predetermined by the server, the server can determine which first file segments failed to upload. For example, if the target file is split into four segments by the client, with corresponding segment numbers 1, 2, 3, and 4, and the server ultimately splits two segments from the target file's streaming data and identifies their segment numbers as 1 and 3, the server can determine, based on the total number of target file segments (i.e., "4"), that segments with sequence numbers 2 and 4 have been successfully uploaded. It should be noted that the aforementioned sequence numbering rule is merely exemplary; any other sequence numbering format that reflects order can also be used, and this disclosure does not impose any limitations.

[0061] Streaming technologies typically use persistent connections to upload files, which are prone to disconnection during periods of network fluctuation. Once a connection is broken, the requesting client often doesn't know how much data the server has received, how to resume the upload, or from which position in the file to resume. To address this, one or more embodiments of this disclosure enable the server to re-slice the continuous streaming data to obtain the first file fragment (or the target data block containing that fragment) obtained from the client's slicing process. This ensures a one-to-one correspondence between the file fragments obtained from the server's slicing process and those obtained from the client's slicing process. By identifying the fragment sequence number in the target data block, the server can ultimately determine which first file fragments of the target file were not successfully uploaded, thus enabling the resume functionality in streaming scenarios.

[0062] In one specific implementation, the server can construct an identifier set with a corresponding number of identifiers based on the total number of file fragments. Each identifier corresponds one-to-one with a first file fragment, and the identifier indicates whether the corresponding first file fragment was successfully uploaded. For example, when an identifier record is 0, it indicates that the first file fragment corresponding to that identifier was not successfully uploaded; conversely, when an identifier record is 1, it indicates that the first file fragment corresponding to that identifier was successfully uploaded. Whenever a file fragment is successfully uploaded, the corresponding identifier in the identifier set can be updated based on its fragment sequence number. In this way, the identifier set can serve as a tool reflecting the specific upload status of the first file fragments. Based on this tool, the server can easily determine which first file fragments were not successfully uploaded and inform the client, enabling the client to continue uploading these unsuccessfully uploaded first file fragments, thereby realizing the file fragment re-upload function.

[0063] In one specific implementation, the identifier set includes a bitmap. A bitmap can be viewed as an array of bits, where each cell can only store 0 or 1. Therefore, using a bitmap to control the success or failure of file fragment transmission to assist in the file resuming function can greatly save storage space.

[0064] In some embodiments, before receiving streaming data of the target file from the client, in response to a first upload request for the target file from the client, the identifier set can be constructed based on the total number of fragments. For example, the first upload request can be a request sent by the client to the server to begin uploading the file before formally transmitting streaming data of the target file to the server. It can carry information related to the target file, such as the file size information of the target file, or the total number of fragments of the first file's data. In this embodiment, by constructing an identifier set for managing the upload status of the target file in advance in response to the first upload request before formally receiving the streaming data of the target file, the upload status of the target file can be managed in a timely manner. In some embodiments, the server can obtain the total number of fragments of the first file's data carried in the first upload request, or determine the total number of fragments based on a predetermined fragment size and the file size of the target file carried in the first upload request.

[0065] Furthermore, some embodiments of the file transfer method also include:

[0066] Step B1: In response to the first upload request, generate a file upload identifier that uniquely corresponds to the target file;

[0067] Step B2: Send the file upload identifier of the target file to the client so that the file upload identifier can be carried in subsequent upload and / or resume upload requests sent by the client for the target file.

[0068] In this embodiment, after receiving the first upload request from the client, the server can return a file upload identifier that uniquely corresponds to the target file to be uploaded. After receiving the file upload identifier, the client can send the file upload identifier to the server when it subsequently uploads or resumes the target file. After receiving the file identifier, the server can identify that the file fragment data uploaded or resumed by the client belongs to the target file.

[0069] Figure 2A schematic diagram of a file transfer system according to an embodiment of this disclosure is shown, including a client 20 and a server 30. The client 20 includes, but is not limited to, mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), digital TVs, desktop computers, etc. The client 20 communicates directly with the server 30 via a local area network, a wide area network, or the Internet, or indirectly via other servers.

[0070] Figure 3 A signal flow diagram between a client 20 and a server 30 in a charging system provided according to an embodiment of this disclosure is shown. (Reference) Figure 3 In step S201, the client 20 performs slicing processing on the target file to be uploaded to obtain first file slice data. For example, the file slice size can be predetermined, and the target file is sliced ​​based on this predetermined file slice size. Similarly, when the server subsequently slices the streaming data, it also slices based on the predetermined file slice size to ensure that the file slice sizes obtained by the server and the client are the same and correspond one-to-one.

[0071] In step S202, the client 20 sends a first file upload request to the server 30. The first file upload request may carry metadata of the target file, such as the target file's file name, file size, permissions, user ID, etc.

[0072] In step S203, the server 30 can generate a file upload identifier that uniquely corresponds to the target file based on the first file upload request.

[0073] In step S204, the server 30 can construct an identifier set (e.g., a bitmap) based on the file size of the target file. For example, the ratio of the target file size to the predetermined file chunk size can be calculated and rounded up to obtain the total number of chunks corresponding to the target file. An identifier set is then constructed based on this total number of chunks, where the number of identifiers in the identifier set is the same as the total number of chunks. Each identifier corresponds one-to-one with each first file chunk data, and each identifier is used to indicate whether the corresponding first file chunk data has been successfully uploaded.

[0074] In addition, after receiving the first file upload request, the server 30 can also verify the client's permissions and store the relevant metadata carried by the request. This embodiment does not impose any restrictions on this.

[0075] In step S205, the server 30 may return a request result in response to the first file upload request to the client 20, and the request result may include the file upload identifier determined by the server.

[0076] In step S206, client 20 can send a formal file upload request to server, establish a long network connection, and stream the data of each first file slice. The streaming data of the target file includes two or more target data blocks, each target data block containing a first file slice data and its corresponding slice sequence number; the slice sequence number is used to indicate the order of the corresponding first file slice data relative to other first file slices in the target file. The formal file upload request may also include the file upload identifier previously returned by server 30.

[0077] In step S207, the server 30 can perform slicing and fragmented storage on the received streaming data. For example, the server 30 can start multiple coroutines or threads to slice the received streaming data, verify the accuracy of the resulting file fragments, encrypt and store them, and record the metadata of the file fragments in the database. The file fragments obtained by the server-side slicing process correspond one-to-one with the file fragments obtained by the client-side slicing process previously.

[0078] In step S208, the server 30 can update the identifier set based on the obtained file fragment data, and determine the first file fragment data that failed to be uploaded based on the identifier set.

[0079] If there is a first file fragment that failed to upload, in step S209, the server 30 can return a request result in response to the formal file upload request to the client 20. The request result may include information indicating the first file fragment that failed to be transmitted to the server (such as the corresponding fragment number or a range of fragment numbers).

[0080] In step S210, client 20 sends a resume request for the target file to the server to resume the first file fragment data that was not successfully uploaded.

[0081] After all the first file fragments of the target file are successfully uploaded, other clients can download the target file from server 30.

[0082] One or more embodiments of this disclosure implement a large file upload method that can be directly applied to SaaS (Software as a Service) business scenarios. This method effectively solves the problem of high-frequency request counts in chunked uploads, optimizes various common problems in streaming transmission, and optimizes the file upload rate of large files.

[0083] Accordingly, refer to Figure 4 According to an embodiment of this disclosure, a file transfer method 400 is provided, applied to a client, including:

[0084] Step S410: Perform slicing on the target file to be transmitted to obtain the first file slice data of the target file;

[0085] Step S420: Send streaming data containing the first file fragment data to the server, so that the server performs the following steps: receiving the streaming data; performing segmentation processing on the streaming data to obtain the file fragment data of the target file; storing the file fragment data of the target file. The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number indicates the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0086] In some embodiments, the client can employ two or more links for concurrent transmission, with each link streaming at least one first file fragment. For example, multiple threads can concurrently stream the first file fragments of the target file, with each thread responsible for transmitting one or more first file fragments. In this embodiment, the target file is divided into multiple file fragments on the client side, and each fragment is identified by a fragment number in the streaming data. This allows the server to know the order of each file fragment, enabling multi-threaded streaming of the target file's file fragments without being affected by the disorder of multi-threaded execution. Furthermore, traditional streaming technologies often use a single long-lived connection for data transmission. This single-connection-based data transmission method is easily affected by network fluctuations, resulting in slow data transmission. In contrast, this embodiment, through concurrent streaming, ensures that other connections can continue transmitting even when one connection is broken, thus efficiently utilizing the user's upload bandwidth.

[0087] In some embodiments, before transmitting the streaming data of the target file to the server, the client may first send a first upload request to the server, which may carry the file size information of the target file to be transmitted, or the total number of fragments of the first file fragment data, so that the server can build the target module in advance.

[0088] In some embodiments, the client may also receive a file upload identifier generated by the server in response to the first file upload request, which is uniquely associated with the target file. This file upload identifier is used to be carried in subsequent upload and / or resume upload requests sent by the client for the target file.

[0089] As for the client-side implementation, since it basically corresponds to the server-side and system-side implementations, the relevant details can be found in the description of the foregoing implementations, and will not be repeated here.

[0090] Accordingly, refer to Figure 5 According to one embodiment of this disclosure, a file transfer device 500 is provided, comprising:

[0091] The receiving unit 501 is used to receive streaming data from the target file of the client;

[0092] The slicing unit 502 is used to slice the streaming data to obtain file fragment data of the target file;

[0093] Storage unit 503 is used to store file fragment data of the target file;

[0094] The streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block contains a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file.

[0095] According to one or more embodiments of this disclosure, by receiving streaming data of a target file from a client, slicing the streaming data to obtain file fragment data of the target file, and storing the file fragment data of the target file, the network link idle time generated during the transmission of the target file can be reduced through streaming transmission. Furthermore, by slicing the received streaming data on the server side, the file fragment data of the target file can be quickly stored and memory reclaimed, thereby reducing the memory usage rate when storing the target file on the server side and avoiding the memory write amplification problem.

[0096] In some embodiments, the streaming data is transmitted concurrently using two or more links, with each link transmitting streaming data formed by at least one first file fragment.

[0097] In some embodiments, the file fragment data of the target file includes the first file fragment data; the file transfer device further includes: a determining unit, configured to determine the first file fragment data that was not successfully transferred to the server based on the fragment sequence number corresponding to the first file fragment data obtained by the server and the total number of fragments of the first file fragment data contained in the target file; and a returning unit, configured to send a preset error message to the client, the error message being used to indicate the first file fragment data that was not successfully transferred to the server.

[0098] In some embodiments, the file transfer device further includes:

[0099] The construction unit is used to construct an identifier set with identifiers based on the total number of fragments, wherein the number of identifiers is the same as the total number of fragments; the identifiers correspond one-to-one with the first file fragment data, and the identifiers are used to indicate whether the corresponding first file fragment data has been successfully uploaded;

[0100] The determining unit is used to update the identifier set based on the fragment sequence number corresponding to the file fragment data obtained from the server, and to determine the file fragment data that was not successfully transmitted based on the identifier set.

[0101] In some embodiments, the identifier set includes a bitmap.

[0102] In some embodiments, the construction unit is configured to construct the identifier set based on the total number of fragments in response to a first upload request for the target file from the client before receiving streaming data of the target file from the client; wherein the total number of fragments is carried in the first upload request, or the total number of fragments is determined based on a predetermined file fragment size and the size of the target file carried in the first upload request.

[0103] In some embodiments, the file transfer device further includes:

[0104] The identifier generation unit is used to generate a file upload identifier that uniquely corresponds to the target file in response to a first upload request for the target file from the client before receiving streaming data of the target file from the client.

[0105] The identifier sending unit is used to send the file upload identifier of the target file to the client, so that the file upload identifier can be carried in the subsequent upload and / or resume upload requests sent by the client for the target file.

[0106] In some embodiments, the target data block further includes target verification information corresponding to the file fragment data; the file transfer device further includes: a verification unit, configured to calculate the target verification information based on the file fragment data of the target file obtained by the server through slicing processing, and compare the calculated target verification information with the target verification information in the target data block to determine whether the file fragment data in the target data block is accurate.

[0107] Accordingly, refer to Figure 6 According to another embodiment of this disclosure, a file transfer device 600 is provided, comprising:

[0108] The slicing unit 601 is used to slice the target file to be transmitted to obtain the first file fragment data of the target file;

[0109] The sending unit 602 is configured to send streaming data containing the first file fragment data to the server, so that the server performs the following steps: receiving the streaming data; performing segmentation processing on the streaming data to obtain the file fragment data of the target file; and storing the file fragment data of the target file.

[0110] The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0111] In some embodiments, the sending unit is used to concurrently transmit streaming data of the target file to the server using two or more links, with each link transmitting streaming data formed by at least one first file fragment.

[0112] In some embodiments, the file transfer device further includes:

[0113] The request unit is configured to send a first upload request for the target file to the server before sending the streaming data to the server. The first upload request carries the size of the target file or the total number of fragments of the first file fragment data.

[0114] In some embodiments, the file transfer device further includes:

[0115] The request unit is used to send a first upload request for the target file to the server, so that the server responds to the first file upload request and generates a file upload identifier that uniquely corresponds to the target file.

[0116] The receiving unit is configured to receive the file upload identifier sent by the server, so that the file upload identifier can be carried in subsequent upload and / or resume requests sent by the client for the target file.

[0117] For embodiments of the apparatus, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments. The apparatus embodiments described above are merely illustrative, and the modules described as separate modules may or may not be separate. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Accordingly, according to one or more embodiments of this disclosure, an electronic device is provided, comprising:

[0119] At least one memory and at least one processor;

[0120] The memory is used to store program code, and the processor is used to call the program code stored in the memory to cause the electronic device to execute the file transfer method provided according to one or more embodiments of this disclosure.

[0121] Accordingly, according to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, which stores program code that can be executed by a computer device to cause the computer device to perform a file transfer method provided according to one or more embodiments of the present disclosure.

[0122] The following is for reference. Figure 7 This illustration shows a structural diagram of an electronic device (e.g., a terminal device or a server) 800 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0123] like Figure 7 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0124] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0125] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0126] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0127] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0128] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0129] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods of the present disclosure.

[0130] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0132] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0133] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0134] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0135] According to one or more embodiments of this disclosure, a file transfer method is provided, applied to a server, comprising: receiving streaming data of a target file from a client; performing slicing processing on the streaming data to obtain file fragment data of the target file; and storing the file fragment data of the target file; wherein the streaming data is generated based on first file fragment data of the target file obtained by the client performing slicing processing on the target file; the streaming data of the target file includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment sequence number; the fragment sequence number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0136] According to one or more embodiments of this disclosure, the streaming data is transmitted concurrently using two or more links, with each link transmitting streaming data formed by at least one first file fragment.

[0137] According to one or more embodiments of this disclosure, the file fragment data of the target file includes the first file fragment data; the provided file transfer method further includes: determining the first file fragment data that was not successfully transferred to the server based on the fragment sequence number corresponding to the first file fragment data obtained by the server and the total number of fragments of the first file fragment data contained in the target file; sending a preset error message to the client, the error message being used to indicate the first file fragment data that was not successfully transferred to the server.

[0138] The file transfer method provided according to one or more embodiments of this disclosure further includes: constructing an identifier set with identifiers based on the total number of fragments, wherein the number of identifiers is the same as the total number of fragments; the identifiers correspond one-to-one with the first file fragment data, and the identifiers are used to identify whether the corresponding first file fragment data has been successfully uploaded; determining the first file fragment data that was not successfully transmitted to the server based on the fragment sequence number corresponding to the first file fragment data obtained from the server and the total number of fragments of the first file fragment data contained in the target file includes: updating the identifier set based on the fragment sequence number corresponding to the file fragment data obtained from the server; and determining the file fragment data that was not successfully transmitted based on the identifier set.

[0139] According to one or more embodiments of this disclosure, the identifier set includes a bitmap.

[0140] According to one or more embodiments of this disclosure, constructing an identifier set with identifiers based on the total number of fragments includes: before receiving streaming data of a target file from a client, in response to a first upload request for the target file from the client, constructing the identifier set based on the total number of fragments; wherein the total number of fragments is carried in the first upload request, or the total number of fragments is determined based on a predetermined file fragment size and the size of the target file carried in the first upload request.

[0141] According to one or more embodiments of this disclosure, before receiving streaming data of a target file from a client, the provided file transfer method further includes: generating a file upload identifier uniquely corresponding to the target file in response to a first upload request for the target file from the client; and sending the file upload identifier of the target file to the client so that the file upload identifier can be carried in subsequent upload and / or resume requests sent by the client for the target file.

[0142] According to one or more embodiments of this disclosure, the target data block further includes target verification information corresponding to the file fragment data; the provided file transfer method further includes: calculating target verification information based on the file fragment data of the target file obtained by the server through slicing processing; comparing the calculated target verification information with the target verification information in the target data block to determine whether the file fragment data in the target data block is accurate.

[0143] According to one or more embodiments of this disclosure, a file transfer method is provided, applied to a client, comprising: performing slicing processing on a target file to be transferred to obtain first file fragment data of the target file; sending streaming data containing the first file fragment data to a server, so that the server performs the following steps: receiving the streaming data; performing slicing processing on the streaming data to obtain file fragment data of the target file; storing the file fragment data of the target file; wherein the streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment sequence number; the fragment sequence number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0144] According to one or more embodiments of this disclosure, sending streaming data containing the first file fragment data to the server includes: concurrently transmitting streaming data of the target file to the server using two or more links, wherein each link streams at least one streaming data formed by the first file fragment data.

[0145] According to one or more embodiments of this disclosure, before sending the streaming data to the server, the provided file transfer method further includes: sending a first upload request to the server for the target file, wherein the first upload request carries the size of the target file or the total number of fragments of the first file fragment data.

[0146] According to one or more embodiments of this disclosure, before sending the streaming data to the server, the provided file transfer method includes: sending a first upload request for the target file to the server, so that the server responds to the first file upload request by generating a file upload identifier uniquely corresponding to the target file; and receiving the file upload identifier sent by the server, so that the file upload identifier can be carried in subsequent upload and / or resume requests sent by the client for the target file.

[0147] According to one or more embodiments of this disclosure, a file transfer apparatus is provided, comprising: a receiving unit for receiving streaming data of a target file from a client; a slicing unit for slicing the streaming data to obtain file fragment data of the target file; and a storage unit for storing the file fragment data of the target file; wherein the streaming data is generated based on first file fragment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment sequence number; the fragment sequence number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0148] According to one or more embodiments of this disclosure, a file transfer apparatus is provided, comprising: a slicing unit for slicing a target file to be transferred to obtain first file fragment data of the target file; and a sending unit for sending streaming data containing the first file fragment data to a server, so that the server performs the following steps: receiving the streaming data; slicing the streaming data to obtain file fragment data of the target file; and storing the file fragment data of the target file; wherein the streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment sequence number; the fragment sequence number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file.

[0149] According to one or more embodiments of the present disclosure, an electronic device is provided, including: at least one memory and at least one processor; wherein the memory is used to store program code, and the processor is used to invoke the program code stored in the memory to cause the electronic device to perform a file transfer method provided according to one or more embodiments of the present disclosure.

[0150] According to one or more embodiments of the present disclosure, a non-transitory computer storage medium is provided, the non-transitory computer storage medium storing program code, which, when executed by a computer device, causes the computer device to perform a file transfer method provided according to one or more embodiments of the present disclosure.

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0152] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A file transfer method, applied on a server side, characterized in that, include: Receive streaming data from the target file in the client; The streaming data is sliced ​​to obtain file fragment data of the target file; Store the file fragment data of the target file; The streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block contains a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file; The file fragment data of the target file obtained by the server-side slicing process corresponds one-to-one with the first file fragment data obtained by the client-side slicing process.

2. The method according to claim 1, characterized in that, Also includes: The streaming data is transmitted concurrently using two or more links, with each link transmitting streaming data formed by at least one first file fragment.

3. The method according to claim 1, characterized in that, The file fragment data of the target file includes the first file fragment data; The method further includes: Based on the fragment sequence number corresponding to the first file fragment data obtained from the server and the total number of fragments of the first file fragment data contained in the target file, the first file fragment data that was not successfully transmitted to the server is determined. A preset error message is sent to the client, the error message indicating that the first file fragment data was not successfully transmitted to the server.

4. The method according to claim 3, characterized in that, The method further includes: constructing an identifier set with identifiers based on the total number of fragments, wherein the number of identifiers is the same as the total number of fragments; the identifiers correspond one-to-one with the first file fragment data, and the identifiers are used to indicate whether the corresponding first file fragment data has been successfully uploaded; The step of determining the first file fragment data that was not successfully transmitted to the server based on the fragment sequence number corresponding to the first file fragment data obtained from the server and the total number of fragments of the first file fragment data contained in the target file includes: updating the identifier set based on the fragment sequence number corresponding to the file fragment data obtained from the server; and determining the file fragment data that was not successfully transmitted based on the identifier set.

5. The method according to claim 4, characterized in that, The set of identifiers includes bitmaps.

6. The method according to claim 4, characterized in that, The construction of an identifier set with identifiers based on the total number of fragments includes: Before receiving streaming data of the target file from the client, in response to a first upload request for the target file from the client, the identifier set is constructed based on the total number of fragments; The total number of file segments is carried in the first upload request, or the total number of file segments is determined based on a predetermined file segment size and the size of the target file carried in the first upload request.

7. The method according to claim 1, characterized in that, Before receiving streaming data from the target file on the client, the method also includes: In response to a first upload request for the target file from the client, a file upload identifier uniquely corresponding to the target file is generated; Send the file upload identifier of the target file to the client so that the file upload identifier can be carried in subsequent upload and / or resume requests sent by the client for the target file.

8. The method according to claim 1, characterized in that, The target data block also includes target verification information corresponding to the file fragment data; The method further includes: Based on the file fragment data of the target file obtained by the server through slicing, the target verification information is calculated; The calculated target verification information is compared with the target verification information within the target data block to determine whether the file fragment data within the target data block is accurate.

9. A file transfer method, applied to a client, characterized in that, include: The target file to be transmitted is sliced ​​to obtain the first file slice data of the target file; Send streaming data containing the first file fragment data to the server so that the server performs the following steps: receive the streaming data; slice the streaming data to obtain the file fragment data of the target file; and store the file fragment data of the target file. The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file; The file fragment data of the target file obtained by the server-side slicing process corresponds one-to-one with the first file fragment data obtained by the client-side slicing process.

10. The method according to claim 9, characterized in that, Sending streaming data containing the first file fragment data to the server includes: The target file is transmitted concurrently to the server using two or more links, with each link transmitting at least one stream of data formed by a first file fragment.

11. The method according to claim 9, characterized in that, Before sending the streaming data to the server, the following steps are also included: Send a first upload request for the target file to the server. The first upload request carries the size of the target file or the total number of fragments of the first file fragment data.

12. The method according to claim 9, characterized in that, Before sending the streaming data to the server, the following steps are also included: Send a first upload request for the target file to the server so that the server responds to the first upload request and generates a file upload identifier that uniquely corresponds to the target file; The file upload identifier sent by the server is received so that it can be carried in subsequent upload and / or resume requests sent by the client for the target file.

13. A file transfer device, characterized in that, include: The receiving unit is used to receive streaming data from the target file of the client; A slicing unit is used to slice the streaming data to obtain file fragment data of the target file; Storage unit, used to store file fragment data of the target file; The streaming data is generated based on the first file segment data of the target file obtained by the client slicing the target file; the streaming data of the target file includes two or more target data blocks, each target data block contains a first file segment data and its corresponding segment number; the segment number is used to indicate the order of the corresponding first file segment data relative to other first file segment data in the target file; The file fragment data of the target file obtained by the server-side slicing process corresponds one-to-one with the first file fragment data obtained by the client-side slicing process.

14. A file transfer device, characterized in that, include: A slicing unit is used to slice the target file to be transmitted to obtain the first file fragment data of the target file; The sending unit is configured to send streaming data containing the first file fragment data to the server, so that the server performs the following steps: receiving the streaming data; performing segmentation processing on the streaming data to obtain the file fragment data of the target file; and storing the file fragment data of the target file. The streaming data includes two or more target data blocks, each target data block containing a first file fragment data and its corresponding fragment number; the fragment number is used to indicate the order of the corresponding first file fragment data relative to other first file fragment data in the target file; The file fragment data of the target file obtained by the server-side slicing process corresponds one-to-one with the first file fragment data obtained by the client-side slicing process.

15. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store program code, and the processor is used to call the program code stored in the memory to cause the electronic device to perform the method of any one of claims 1 to 12.

16. A non-transitory computer storage medium, characterized in that, The non-transitory computer storage medium stores program code, which, when executed by a computer device, causes the computer device to perform the method according to any one of claims 1 to 12.

Images