A file uploading method and related apparatus

Abstract

The application discloses a file uploading method and related device, and relates to the technical field of software, and comprises the following steps: acquiring a target file, the target file being a to-be-uploaded file meeting large file screening conditions; in response to an uploading operation of the target file, dividing the target file into continuous multiple slice units; taking the multiple slice units as uploading objects, determining a first slice unit to be uploaded in a first direction and a second slice unit to be uploaded in a second direction, the first direction and the second direction being opposite directions; according to a historical network speed measured at a previous time, dividing a first sub-slice of a corresponding slice size from the first slice unit for uploading, simultaneously, dividing a second sub-slice of a specified slice size from the second slice unit for uploading, and measuring a current network speed at a current time according to an uploading result of the second sub-slice, the current network speed being used for division and uploading of sub-slices in the first slice unit at a next time. Therefore, efficient file uploading in a complex network environment is realized.

Description

Technical Field

[0001] This application relates to the field of software technology, and in particular to a file upload method and related apparatus. Background Technology

[0002] With the widespread adoption of the internet and the rapid growth of data volume, users' demand for file uploads is increasing. When faced with large files such as high-definition videos, image datasets, and large documents, traditional file upload methods typically rely on fixed-chunk transmission, which cannot adapt to complex network environments, resulting in slow upload speeds and high failure rates. Summary of the Invention

[0003] In view of the above problems, this application provides a file upload method and related apparatus to achieve efficient file upload in complex network environments. The specific solution is as follows:

[0004] The first aspect of this application provides a file upload method, the file upload method comprising:

[0005] Obtain the target file, which is a file to be uploaded that meets the large file filtering criteria;

[0006] In response to the upload operation of the target file, the target file is divided into multiple consecutive fragment units;

[0007] The plurality of fragmented units are used as upload objects. The upload objects are determined to be the first fragmented unit to be uploaded in the first direction and the second fragmented unit to be uploaded in the second direction, wherein the first direction and the second direction are opposite directions.

[0008] Based on the historical network speed measured at the previous time, a first sub-fraction of the corresponding size is divided from the first fragmentation unit and uploaded. At the same time, a second sub-fraction of the specified size is divided from the second fragmentation unit and uploaded. The current network speed at the current time is measured based on the upload result of the second sub-fraction. The current network speed is used for the division and uploading of sub-fractions in the first fragmentation unit at the next time.

[0009] In one possible implementation, uploading the first sub-fragment, which is divided into corresponding fragment sizes from the first fragmentation unit based on the historical network speed measured at the previous time, includes:

[0010] Obtain the shard size configuration, which includes the maximum shard size and the minimum shard size;

[0011] The segment size threshold for the slow start phase is determined based on the historical network speed, the preset security factor, and the preset ideal upload time.

[0012] Obtain the size of the third sub-fragment that was most recently successfully uploaded in the first direction, and compare the size of the third sub-fragment with the fragment size threshold;

[0013] If the size of the third sub-segment is less than the segment size threshold, the size of the third sub-segment is exponentially increased to obtain the candidate segment size of the first sub-segment;

[0014] If the size of the third sub-segment is greater than or equal to the segment size threshold, the segment size of the third sub-segment is linearly increased according to the minimum segment size to obtain the candidate segment size of the first sub-segment;

[0015] The dynamic fragment size of the first sub-fragment is determined based on the historical network speed and the ideal upload time;

[0016] The minimum value among the maximum fragment size, the candidate fragment size, and the dynamic fragment size is used as the target fragment size of the first sub-fragment, and the target fragment size is used as the basis for the division and uploading of the first sub-fragment.

[0017] In one possible implementation, the step of uploading a first sub-fragment of the corresponding fragment size from the first fragmentation unit based on the historical network speed measured at the previous time also includes:

[0018] If the first sub-part fails to upload, the minimum sub-part size is taken as the target sub-part size, and the cumulative number of uploads of the first sub-part is counted.

[0019] If the cumulative upload count does not reach the preset maximum upload count, wait to upload the first sub-part until other first part units in the first direction are successfully uploaded, then re-upload the first sub-part, and increment the cumulative upload count by 1.

[0020] If the cumulative number of uploads reaches the preset maximum number of uploads, the first segment unit is determined to have failed to upload.

[0021] In one possible implementation, measuring the current network speed at the current time based on the upload result of the second sub-segment includes:

[0022] Get the upload time of the second sub-part;

[0023] The current instantaneous network speed at the current time is determined based on the specified fragment size and the upload time.

[0024] The current instantaneous network speed is stored as a data point in a sliding window, and the current smooth network speed is calculated after outlier removal from all data points in the sliding window.

[0025] In one possible implementation, measuring the current network speed at the current time based on the upload result of the second sub-segment further includes:

[0026] Obtain the historical average network speed, which is the average of the historical instantaneous network speed over a historical period;

[0027] The current converged network speed is calculated based on the historical average network speed and the current smoothed network speed.

[0028] In one possible implementation, the upload task of the first fragment unit is a first upload task, and the upload task of the second fragment unit is a second upload task. The file upload method further includes:

[0029] Initialize a thread pool, and create multiple concurrent threads in the thread pool. Among the multiple threads, the first thread is used to execute the first upload task, and the second thread is used to execute the second upload task. There is at least one first thread and one second thread, and the first thread corresponds one-to-one with the first upload task.

[0030] The first thread is invoked to execute the first upload task, so as to divide the first sub-segment from the first segmentation unit and upload it; and the second thread is invoked to execute the corresponding second upload task, so as to divide the second sub-segment from the second segmentation unit and upload it, and measure the current network speed.

[0031] Monitor the task execution status of the first thread;

[0032] If the first sub-fragment under the first target thread in the first thread is successfully uploaded, a new first thread is created in the thread pool.

[0033] If the first sub-fragment under the second target thread in the first thread is waiting to be uploaded, stop running the second target thread until the first fragment unit under the third target thread in the first thread is successfully uploaded, then restart the second target thread.

[0034] If the first fragment unit under the fourth target thread in the first thread fails to upload, stop running the fourth target thread and add the fourth target thread to the failure queue;

[0035] If the first shard unit under the fifth target thread in the first thread is successfully uploaded, stop running the fifth target thread and release the resources of the fifth target thread.

[0036] In one possible implementation, the file upload method further includes:

[0037] In response to the breakpoint resume operation of the target file, at least one target fragment unit in the target file that has not been uploaded is identified;

[0038] The at least one target fragment unit is taken as the upload object, and the process returns to the step of determining the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction.

[0039] A second aspect of this application provides a file uploading device, the file uploading device comprising:

[0040] The file acquisition module is used to acquire the target file, which is a file to be uploaded that meets the large file filtering criteria;

[0041] The file upload module is used to respond to the upload operation of the target file by dividing the target file into multiple consecutive fragment units; using the multiple fragment units as upload objects, determining the first fragment unit to be uploaded in a first direction and the second fragment unit to be uploaded in a second direction, wherein the first direction and the second direction are opposite directions; uploading the first sub-fragments of the corresponding fragment size from the first fragment unit according to the historical network speed measured at the previous time; simultaneously uploading the second sub-fragments of the specified fragment size from the second fragment unit; and measuring the current network speed at the current time based on the upload result of the second sub-fragments, wherein the current network speed is used for the division and uploading of sub-fragments in the first fragment unit at the next time.

[0042] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the file upload method described in the first aspect or any implementation thereof.

[0043] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0044] The memory is used to store computer programs;

[0045] The processor is used to execute the computer program so that the electronic device can implement the file upload method described in the first aspect or any implementation thereof.

[0046] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the file upload method described in the first aspect or any implementation thereof.

[0047] Using the above technical solution, this application provides a file upload method and related apparatus, comprising: acquiring a target file, wherein the target file is a file to be uploaded that meets the large file filtering conditions; in response to the upload operation of the target file, dividing the target file into multiple consecutive fragment units; using the multiple fragment units as upload objects, determining the first fragment unit to be uploaded in a first direction and the second fragment unit to be uploaded in a second direction, wherein the first direction and the second direction are opposite directions; uploading a first sub-fraction of a corresponding fragment size from the first fragment unit according to the historical network speed measured at the previous time, and simultaneously uploading a second sub-fraction of a specified fragment size from the second fragment unit, and measuring the current network speed at the current time according to the upload result of the second sub-fraction, wherein the current network speed is used for the division and uploading of sub-fractions in the first fragment unit at the next time. This application divides large files into contiguous chunks during upload. Two chunks are uploaded in opposite directions. One chunk is dynamically divided into sub-chunks based on historical network speeds from the previous time period, while the other chunk is divided into sub-chunks of a specified size and the current network speed is measured for subsequent sub-chunk division. The rationale behind this application is that the independent chunking upload mechanism improves transmission efficiency and stability. Furthermore, to adapt to changing network environments, real-time network speed measurement and dynamic adjustment of chunk sizes ensure full utilization of upload bandwidth, thereby improving upload speed and success rate. Attached Figure Description

[0048] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0049] Figure 1 A flowchart illustrating a file upload method provided in an embodiment of this application;

[0050] Figure 2 This application provides an example of a segmentation unit;

[0051] Figure 3 This is a partial flowchart illustrating a file upload method provided in an embodiment of this application;

[0052] Figure 4 This is another schematic flowchart of a file upload method provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of a file upload device provided in an embodiment of this application;

[0054] Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0056] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0057] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0058] To address the problems of low upload speed, high failure rate, and inability to resume interrupted uploads often encountered by existing file upload technologies when facing network instability, bandwidth fluctuations, and upload interruptions, this application provides a file upload method. By introducing file fragmentation transmission, dynamic fragmentation adjustment, multi-threaded concurrent upload, failure retry mechanism, and interrupted upload capability, it offers an efficient, robust, and flexible method for uploading large files. This improves upload efficiency and reliability in variable network environments, optimizes bandwidth utilization, and simplifies task management logic. The file upload method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0059] See Figure 1 , Figure 1 This is a flowchart illustrating a file upload method provided in an embodiment of this application. Figure 1 As shown in the figure, the file upload method provided in this application embodiment is applied to a client and may include steps S101 to S104, which are described in detail below.

[0060] S101, Obtain the target file. The target file is the file to be uploaded that meets the large file filtering criteria.

[0061] In this embodiment, the client can select a file to be uploaded and determine whether the file is larger than twice the maximum chunk size. If the file is larger than twice the maximum chunk size, it is determined that it meets the large file filtering criteria, and the file is used as the target file for subsequent processes. If the file is smaller than or equal to twice the maximum chunk size, it is determined that it does not meet the large file filtering criteria, and the file is divided into a fixed number of chunks and uploaded using a conventional chunked transmission method.

[0062] In practical applications, after obtaining the target file, the client can generate a unique file identifier based on the file content, for example, by using an MD5 or SHA256 hash algorithm to calculate the file identifier. This file identifier can be used for subsequent upload verification and file deduplication. Of course, the unique identifiers of subsequent allocation units and sub-fragments can also be calculated using MD5 or SHA256 hash algorithms; this application embodiment does not limit this.

[0063] S102, in response to the upload operation of the target file, divides the target file into multiple consecutive fragment units.

[0064] In this embodiment, the client can respond to the upload operation input for the target file and divide the target file into multiple contiguous chunks according to the maximum chunk size. The size of each chunk is the maximum chunk size. See also Figure 2 , Figure 2 This application provides an example of a segmentation unit partitioning scheme. For example... Figure 2 As shown, a complete target file is divided into n chunks, each chunk being the largest chunk size. That is, a target file is divided into n consecutive chunks from chunk_0 to chunk_n-1.

[0065] In practical applications, the client can send the file identifier of the target file to the server to query the upload status of the target file. If the server has already stored the target file, it returns the storage path (or access URL) and file completion status to the client. When the file completion status is "complete file," the client can determine that the target file has been uploaded successfully. When the file completion status is "partial file missing," the client enters the breakpoint resume logic and re-uploads the missing file content. If the server has not stored the target file, it returns a status code indicating that the file has not been uploaded to the client. The client then performs the upload operation based on this status code, dividing the target file into multiple consecutive fragment units. The fragment number and offset information of each fragment unit are recorded in the fragment object of that fragment unit for subsequent task management.

[0066] S103, multiple fragment units are used as upload objects, and the upload objects are determined as the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction, wherein the first direction and the second direction are opposite directions.

[0067] In this embodiment of the application, see again Figure 2 Using n chunk units as the upload objects, the chunk units used for subsequent dynamic sub-slicing (i.e., the first chunk unit) are determined sequentially in the first direction, and the chunk units used for subsequent fixed sub-slicing and speed measurement (i.e., the second chunk unit) are determined sequentially in the second direction. The first direction can be from chunk_0 to chunk_n-1, and the second direction can be from chunk_n-1 to chunk_0.

[0068] S104. Based on the historical network speed measured in the previous time period, the first sub-fraction of the corresponding fragment size is divided from the first fragment unit and uploaded. At the same time, the second sub-fraction of the specified fragment size is divided from the second fragment unit and uploaded. Based on the upload result of the second sub-fraction, the current network speed at the current time period is measured. The current network speed is used for the division and upload of sub-fractions in the first fragment unit in the next time period.

[0069] In this embodiment, for the first fragment unit determined in the first direction, the client can determine the fragment size of the sub-fragment (i.e., the first sub-fragment) to be divided based on the historical network speed measured at a previous time. The historical network speed is positively correlated with the fragment size; that is, the larger the historical network speed, the larger the fragment size of the first sub-fragment. The first sub-fragment is then divided from the first fragment unit according to this fragment size. In practical applications, the first fragment unit uploaded initially divides the first sub-fragment into the smallest fragment size (e.g., 100KB) for uploading. After each upload is completed, the first sub-fragment for the current upload is dynamically divided based on the historical network speed measured at a previous time for the second fragment unit. The offset and range of the first sub-fragment is offset1 = chunk.offset1 + offset.last + taskSize1; where chunk.offset1 represents the start position of the first fragment unit, offset.last represents the end position of the previous first sub-fragment, and taskSize represents the fragment size determined based on the historical network speed. Of course, after a first fragment unit is uploaded, the client can continue to determine new first fragment units to be uploaded in the first direction until all fragment units of the target file have been uploaded.

[0070] For the second fragment unit determined in the second direction, the client can divide its sub-fragments (i.e., the second sub-fragments) according to a specified fragment size (e.g., 100KB). This avoids interference with the speed measurement results due to changes in fragment size. The client then uploads the second sub-fragment and measures the current network speed based on the response time and packet loss rate during the upload. This current network speed is used for the division and upload of the new first sub-fragment in the first fragment unit in the next time period. The network speed measured by the client can accurately reflect the current network environment and provide a reference for dynamic adjustment. The offset and range of the second sub-fragment is offset2 = chunk.offset2 + chunk.length - (taskIndex + 1) × taskSize2; where chunk.offset2 represents the starting position of the second fragment unit, chunk.length represents the size of the second fragment unit, taskIndex represents the index of the previous second sub-fragment, and taskSize2 represents the specified fragment size. Of course, after a second fragment unit is uploaded, the client can continue to determine new second fragment units to be uploaded in the second direction until all fragment units of the target file have been uploaded.

[0071] In practical applications, the client can record the chunk data structure of the target file locally, which includes file identifier, upload status, number of chunk units, total file size, chunk objects of completed upload chunk units, chunk objects of incomplete upload chunk units, upload progress percentage, remaining server retention time, error information, etc. Specifically, the chunk objects of completed upload chunk units contain information about that chunk unit (such as index, unique identifier, starting position, total size, uploaded size, etc.); the chunk objects of incomplete upload chunk units contain information about that chunk unit (such as index, unique identifier, starting position, total size, uploaded size, etc.) and a list of sub-chunks (such as sub-chunk sequence number, unique identifier, starting position, chunk size, etc.). Based on this fragmented data structure, on the one hand, the client can notify the server to merge the fragment unit after each fragment unit's sub-fragments are successfully uploaded. The server can verify the integrity of its sub-fragments based on the unique identifier of the fragment unit. If the sub-fragments are complete, the server will merge the fragment unit; otherwise, if they are incomplete, the server will notify the client to upload the missing sub-fragments. On the other hand, the client can also notify the server to merge the files after all fragment units are successfully uploaded. The server can merge the fragment units sequentially according to their offsets into a complete target file. After merging, the server will calculate the file identifier of the target file and match it with the file identifier provided by the client to ensure data integrity. If the match is successful, the server will return a successful upload status to the client and provide the storage path (or access URL) of the target file. Otherwise, if the match fails, the server will notify the client to upload the missing fragment units.

[0072] In one possible implementation, by dynamically adjusting the size of the first fragment unit to be uploaded, it is possible to adapt to different network conditions (bandwidth, latency, stability), maximizing bandwidth utilization while reducing upload failure rate and resource waste. See also Figure 3 , Figure 3 This is a partial flowchart illustrating a file upload method provided in an embodiment of this application. Figure 3 As shown in the embodiment of this application, a file upload method is provided, wherein step S104, "uploading a first sub-fraction of the corresponding fragment size from the first fragment unit according to the historical network speed measured at the previous time", may include steps S201 to S207, which are described in detail below.

[0073] S201, Get the shard size configuration, which includes the maximum shard size and the minimum shard size.

[0074] In this embodiment of the application, the client can obtain the fragment size configuration input by the user, which includes at least a maximum fragment size and a minimum fragment size. The maximum fragment size is used to limit the upper limit of the fragment size, which is usually the maximum value supported by the server (e.g., 4MB); the minimum fragment size (e.g., 64KB) is used for recovery or speed testing tasks after failure.

[0075] S202 determines the segment size threshold for the slow start phase based on historical network speed, preset security factor, and preset ideal upload time.

[0076] In this embodiment, a smaller fragment size is used in the initial upload phase to test bandwidth capacity, and the fragment size is gradually increased. The client can determine the fragment size threshold for the slow start phase based on historical network speeds, and this threshold can be calculated using the following formula (1):

[0077] ; (1)

[0078] in, Indicates the slice size threshold; This represents the preset safety factor, typically set to 0.85. Indicates historical network speed; This indicates the preset ideal upload time, which is usually set to 300ms.

[0079] S203, obtain the size of the third sub-fragment that was most recently successfully uploaded in the first direction, and compare the size of the third sub-fragment with the fragment size threshold.

[0080] In this embodiment, the client obtains the size of the most recently successfully uploaded sub-fragment (i.e., the third sub-fragment) in the first direction. This third sub-fragment may be located in the same first fragment unit as the currently to-be-divided sub-fragment, or it may be the previously successfully uploaded first fragment unit; this embodiment does not limit this. After obtaining the size of the third sub-fragment, the size relationship between the size of the third sub-fragment and the fragment size threshold can be compared.

[0081] S204, if the size of the third sub-segment is less than the segment size threshold, the size of the third sub-segment is exponentially increased to obtain the candidate segment size of the first sub-segment.

[0082] In this embodiment of the application, if the size of the third sub-segment is less than the segment size threshold... The client uses the following formula (2) to exponentially increase the size of the third sub-partition to obtain the candidate sub-partition size:

[0083] ; (2)

[0084] in, This indicates the candidate fragment size for the first sub-fragment; Indicates the size of the third sub-part; This represents the growth factor.

[0085] S205, if the size of the third sub-segment is greater than or equal to the segment size threshold, the size of the third sub-segment is linearly increased based on the minimum segment size to obtain the candidate segment size of the first sub-segment.

[0086] In this embodiment of the application, if the size of the third sub-segment is greater than or equal to the segment size threshold... The client obtains the candidate fragment size of the first sub-fragment by linearly increasing the fragment size of the third sub-fragment according to the following formula (3):

[0087] ; (3)

[0088] in, This indicates the minimum fragment size.

[0089] S206 determines the dynamic fragment size of the first sub-fragment based on historical network speed and ideal upload time.

[0090] In this embodiment of the application, the client calculates the dynamic fragment size of the first sub-fragment according to the following formula (4):

[0091] ; (4)

[0092] in, This indicates the dynamic fragment size of the first sub-fragment.

[0093] S207: The minimum value among the maximum fragment size, candidate fragment size, and dynamic fragment size is used as the target fragment size for the first sub-fragment. The target fragment size is used as the basis for the division and uploading of the first sub-fragment.

[0094] In this embodiment of the application, the client determines the target fragment size of the first sub-fragment according to the following formula (5), and the target fragment size is the basis for the client to subsequently divide the first sub-fragment and upload it:

[0095] ; (5)

[0096] in, Indicates the target fragment size of the first sub-fragment; This indicates the maximum fragment size.

[0097] In one possible implementation, if the first sub-fragment upload fails (e.g., due to network interruption or verification failure), its fragment size is quickly rolled back to reduce the upload failure rate. The normal upload size is then gradually restored subsequently. Alternatively, a retry mechanism can be used to re-upload after an upload failure. In this regard, an embodiment of this application provides a file upload method, wherein step S104, "uploading the first sub-fragment of the corresponding fragment size from the first fragment unit based on the historical network speed measured at the previous time," may further include the following steps:

[0098] If the first sub-fragment upload fails, the minimum fragment size is used as the target fragment size, and the cumulative upload count of the first sub-fragment is counted. If the cumulative upload count does not reach the preset maximum upload count, the first sub-fragment is waited to be uploaded until other first fragment units under the first direction are successfully uploaded. The first sub-fragment is then re-uploaded, and the cumulative upload count is incremented by 1. If the cumulative upload count reaches the preset maximum upload count, the first fragment unit upload is determined to have failed.

[0099] In this embodiment of the application, if the first sub-part fails to upload, the smallest sub-part size is used as the target sub-part size for partitioning, and the cumulative number of uploads of the first sub-part is counted.

[0100] If the cumulative upload count for the first sub-part is less than the preset maximum upload count, the upload of the first sub-part is paused, and the upload will resume once other first sub-part units in the first direction have successfully uploaded. Simultaneously, the cumulative upload count is incremented by 1. In other words, if an upload task for a certain first sub-part fails, it can wait for the upload tasks of other first sub-part units to succeed before resuming the upload, thereby reducing resource contention between upload tasks.

[0101] In addition, if the cumulative number of uploads to the first sub-segment is equal to the preset maximum number of uploads, it is determined that the upload of the first segment unit to which it belongs has failed, and the upload task of the first segment unit is moved to the failure queue for subsequent analysis or manual retry.

[0102] In one possible implementation, instantaneous network speed can be used to assess current network performance, and by identifying outliers, the impact of network jitter on upload efficiency can be reduced, resulting in a stable and reliable network speed. This allows for dynamic optimization of fragment size, thereby improving upload success rate. See also Figure 4 , Figure 4 This is another schematic flowchart illustrating a file upload method provided in an embodiment of this application. Figure 4 As shown in the embodiment of this application, a file upload method is provided, wherein step S104, "measuring the current network speed at the current time based on the upload result of the second sub-fragment", may include steps S301 to S303, which are described in detail below.

[0103] S301, Get the upload time of the second sub-segment.

[0104] In this embodiment of the application, the client periodically uploads the second sub-fragment and records the upload time of the second sub-fragment.

[0105] S302 determines the current instantaneous network speed at the current time based on the specified fragment size and upload time.

[0106] In this embodiment of the application, the client calculates the current instantaneous network speed at the current time according to the following formula (6):

[0107] ; (6)

[0108] in, Indicates the current instantaneous network speed; Indicates the specified fragment size; This indicates the upload time.

[0109] S303 stores the current instantaneous network speed as a data point in the sliding window, and calculates the current smooth network speed after removing outliers from all data points in the sliding window.

[0110] In this embodiment, the client stores the current instantaneous network speed as data points in a sliding window in ascending order of time. For example, the sliding window size can be the most recent 10 seconds or 100 data points. To avoid interference from individual abnormal network speeds (such as extremely high or low speeds due to network jitter), outliers in the sliding window can be removed using the interquartile range (IQR). Specifically, the data points in the sliding window can be sorted in ascending order according to their numerical values, and the 25th percentile (Q1) and 75th percentile (Q3) can be calculated. The IQR is then calculated as follows (IQR = Q3 - Q1). Data points in the sliding window that are lower than Q1 - 1.5 × IQR or higher than Q3 + 1.5 × IQR are removed as outliers.

[0111] After removing outliers from all data points under the sliding window, the weighted moving average (WMA) algorithm can be used to calculate the current smooth network speed for the remaining data points. Specifically, it can be calculated according to the following formula (7):

[0112] ; (7)

[0113] in, Indicates the current smooth network speed; This indicates the number of data points remaining in the sliding window; Indicates the first in the sliding window The data weight of each data point is determined by its proximity to the current time; for example, the data weight of the 11th data point within the sliding window is 11. Indicates the first in the sliding window The instantaneous network speed of each data point. In this embodiment of the application, the current smoothed network speed can be used as the current network speed.

[0114] In one possible implementation, to further improve the stability of the network speed calculation results, historical average network speeds can be dynamically fused to reduce the impact of network fluctuations. In this regard, an embodiment of this application provides a file upload method, wherein step S104, "measuring the current network speed at the current time based on the upload result of the second sub-fragment," further includes the following steps:

[0115] Obtain the historical average network speed, which is the average of the historical instantaneous network speed over a historical period; calculate the current converged network speed based on the historical average network speed and the current smoothed network speed.

[0116] In this embodiment of the application, the client can obtain the historical instantaneous network speed within a historical time period (e.g., the most recent 10 minutes) and calculate the current converged network speed according to the following formula (8):

[0117] ; (8)

[0118] in, Indicates the current converged network speed; This indicates the fusion weighting coefficient (e.g., 0.7), which controls the weight ratio between smoothed data and historical data to reduce the impact of network fluctuations. This represents the historical average network speed. In this embodiment of the application, the current converged network speed can be used as the current network speed.

[0119] In one possible implementation, a thread pool can be used to manage multi-threaded concurrency, adjusting the number of threads to adapt to network conditions. In this regard, an embodiment of this application provides a file upload method, wherein the upload task of the first segment unit is a first upload task, the upload task of the second segment unit is a second upload task, and the file upload method provided in this embodiment of the application further includes the following steps:

[0120] Initialize a thread pool, creating multiple concurrent threads within it. One thread executes the first upload task, and a second thread executes the second upload task. There is at least one first thread and one second thread, with a one-to-one correspondence between the first thread and the first upload task. Initiate the first thread to execute the first upload task, which involves partitioning the first sub-segment from the first segment unit and uploading it. Then, initiate the second thread to execute the corresponding second upload task, which involves partitioning the second sub-segment from the second segment unit, uploading it, and measuring the current network speed. Monitor the task execution status of the first thread; if the first thread is executing the first target... If the first sub-fragment under a thread is successfully uploaded, a new first thread is created in the thread pool. If the first sub-fragment under the second target thread in the first thread is waiting to be uploaded, the second target thread stops running until the first fragment unit under the third target thread in the first thread is successfully uploaded, then the second target thread is restarted. If the first fragment unit under the fourth target thread in the first thread fails to upload, the fourth target thread stops running and is added to the failure queue. If the first fragment unit under the fifth target thread in the first thread is successfully uploaded, the fifth target thread stops running and its resources are released.

[0121] In this embodiment, before starting the upload task, the client initializes a thread pool and creates multiple concurrent threads in the thread pool, including a first thread for executing a first upload task and a second thread for executing a second upload task. There are one or more first threads and one second thread, with one first thread corresponding to one first upload task. This allows for the simultaneous upload of at least one first fragment unit and one second fragment unit.

[0122] The client invokes each first thread to execute its corresponding first upload task, in order to divide the first sub-segment from the corresponding first segment unit and upload it; similarly, it invokes the second thread to execute the second upload task, in order to divide the second sub-segment from the second segment unit and upload it, and measure the current network speed.

[0123] The client monitors the task execution status of the first thread. If the first sub-segment under a certain thread (i.e. the first target thread) is successfully uploaded, the task status of the first sub-segment is updated in the locally stored segment data structure (from uploading to uploading successfully), the upload time and the return information from the server (such as the segment confirmation flag) are recorded, a new first thread is created in the thread pool, and a new first segment unit is allocated to the new first thread in the first direction. The new first thread then executes the first upload task of the corresponding first segment unit.

[0124] Additionally, if the first shard unit under a certain thread (i.e., the second target thread) is waiting to be uploaded, the task status of the first shard unit is updated in the locally stored shard data results (updated value during upload, waiting to be uploaded), and the second target thread stops running until the first shard unit under another thread (i.e., the third target thread) is successfully uploaded. Then, the second target thread is restarted, reallocated to the thread pool, and the upload process is resumed. At this time, the third target thread stops running and releases its resources.

[0125] Additionally, if the first shard unit uploaded by a certain thread (i.e., the fourth target thread) fails, the task status of that first shard unit is updated in the locally stored shard data results (updating from the failed upload value), the fourth target thread is stopped, and the fourth target thread is added to the failure queue. Subsequently, when a new thread successfully uploads its first shard unit and no thread has stopped running, the fourth target thread is restarted, reallocated to the thread pool, and the upload process resumes.

[0126] Additionally, if the first shard unit under a certain thread (i.e. the fifth target thread) is successfully uploaded, the task status of the first shard unit is updated in the locally stored shard data results (upload success value is updated during upload), the fifth target thread is stopped, and the resources of the fifth target thread are released.

[0127] In one possible implementation, to ensure that the upload can continue even after an interruption, a breakpoint resume logic can be employed to avoid wasting time and resources. In this regard, an embodiment of this application provides a file upload method that further includes the following steps:

[0128] In response to the breakpoint resume operation of the target file, at least one target fragment unit in the target file that has not been uploaded is identified; at least one target fragment unit is taken as the upload object, and the process returns to the step of determining the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction.

[0129] In this embodiment, after each successful upload of a sub-fragment of a fragment unit, the client updates the locally stored fragment data structure. If the upload is interrupted, the client obtains the file completion status returned by the server based on the text identifier (specifically, some files are missing). At this time, the client performs a resume operation, determines at least one fragment unit (i.e., the target fragment unit) in the target file that has not been uploaded from the locally stored fragment data structure, takes all the unuploaded parts in the at least one target fragment unit as the upload object, and returns to the execution of step S103, "determine the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction", and re-uploads the remaining part of the target file, which greatly improves efficiency and reduces the risk of repeated uploads.

[0130] Based on the above description, the file upload method provided in this application, by introducing file fragmentation technology, real-time network speed monitoring, dynamic adjustment of fragment size, multi-threaded concurrent upload, failure retry mechanism, and breakpoint resumption capability, ensures data integrity, reduces resource waste, and achieves efficient, secure, and stable file uploads in complex network environments, while optimizing user experience and server performance. This application is particularly suitable for scenarios where uploads fail due to complex network environments, and has significant practicality and innovation.

[0131] The above describes a file upload method provided by an embodiment of this application. The following describes the apparatus for performing the above file upload method.

[0132] See Figure 5 , Figure 5 This is a schematic diagram of a file upload device provided in an embodiment of this application. Figure 5 As shown in the embodiment of this application, a file upload device includes:

[0133] The file acquisition module 401 is used to acquire the target file, which is a file to be uploaded that meets the large file filtering criteria.

[0134] The file upload module 402 is used to respond to the upload operation of the target file by dividing the target file into multiple consecutive fragment units; taking the multiple fragment units as upload objects, determining the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction, where the first and second directions are opposite directions; uploading the first sub-fragment of the corresponding fragment size from the first fragment unit according to the historical network speed measured at the previous time; simultaneously, uploading the second sub-fragment of the specified fragment size from the second fragment unit; and measuring the current network speed at the current time based on the upload result of the second sub-fragment, which is used for the division and upload of sub-fragments in the first fragment unit at the next time.

[0135] In one possible implementation, the file upload module 402, used to upload a first sub-fragment of corresponding fragment size from the first fragment unit based on the historical network speed measured at the previous time, is specifically used for:

[0136] The process involves: obtaining the fragment size configuration, including the maximum and minimum fragment sizes; determining the fragment size threshold for the slow start phase based on historical network speed, a preset security factor, and a preset ideal upload time; obtaining the fragment size of the third sub-fragment that was most recently successfully uploaded in the first direction and comparing it with the fragment size threshold; if the fragment size of the third sub-fragment is less than the fragment size threshold, exponentially increasing the fragment size of the third sub-fragment to obtain the candidate fragment size of the first sub-fragment; if the fragment size of the third sub-fragment is greater than or equal to the fragment size threshold, linearly increasing the fragment size of the third sub-fragment based on the minimum fragment size to obtain the candidate fragment size of the first sub-fragment; determining the dynamic fragment size of the first sub-fragment based on historical network speed and ideal upload time; and using the minimum value among the maximum fragment size, candidate fragment size, and dynamic fragment size as the target fragment size of the first sub-fragment, which serves as the basis for the partitioning and uploading of the first sub-fragment.

[0137] In one possible implementation, the file upload module 402, used to divide the first sub-fragment of corresponding fragment size from the first fragment unit based on the historical network speed measured at the previous time, is further used for:

[0138] If the first sub-fragment upload fails, the minimum fragment size is used as the target fragment size, and the cumulative upload count of the first sub-fragment is counted. If the cumulative upload count does not reach the preset maximum upload count, the first sub-fragment is waited to be uploaded until other first fragment units under the first direction are successfully uploaded. The first sub-fragment is then re-uploaded, and the cumulative upload count is incremented by 1. If the cumulative upload count reaches the preset maximum upload count, the first fragment unit upload is determined to have failed.

[0139] In one possible implementation, the file upload module 402, used to measure the current network speed at the current time based on the upload result of the second sub-fragment, is specifically used for:

[0140] Get the upload time of the second sub-segment; determine the current instantaneous network speed at the current time based on the specified segment size and upload time; store the current instantaneous network speed as a data point in the sliding window, and calculate the current smooth network speed after removing outliers from all data points in the sliding window.

[0141] In one possible implementation, the file upload module 402, used to measure the current network speed at the current time based on the upload result of the second sub-fragment, is specifically used for:

[0142] Obtain the historical average network speed, which is the average of the historical instantaneous network speed over a historical period; calculate the current converged network speed based on the historical average network speed and the current smoothed network speed.

[0143] In one possible implementation, the upload task of the first segment unit is the first upload task, and the upload task of the second segment unit is the second upload task. The file upload module 402 is also used for:

[0144] Initialize the thread pool and create multiple concurrent threads in the thread pool. Among the multiple threads, the first thread is used to execute the first upload task, the second thread is used to execute the second upload task, there is at least one first thread and one second thread, and the first thread corresponds one-to-one with the first upload task.

[0145] The first thread is invoked to execute the first upload task, which involves dividing the first sub-fragment from the first fragment unit and uploading it; and the second thread is invoked to execute the corresponding second upload task, which involves dividing the second sub-fragment from the second fragment unit and uploading it, and measuring the current network speed.

[0146] Monitor the task execution status of the first thread;

[0147] If the first sub-fragment under the first target thread in the first thread is successfully uploaded, a new first thread is created in the thread pool;

[0148] If the first sub-fragment under the second target thread in the first thread is waiting to be uploaded, stop running the second target thread until the first fragment unit under the third target thread in the first thread is successfully uploaded, then restart the second target thread.

[0149] If the first fragment unit under the fourth target thread in the first thread fails to upload, stop running the fourth target thread and add the fourth target thread to the failure queue;

[0150] If the first fragment unit under the fifth target thread in the first thread is successfully uploaded, stop running the fifth target thread and release the resources of the fifth target thread.

[0151] In one possible implementation, the file upload module 402 is also used for:

[0152] In response to the breakpoint resume operation of the target file, at least one target fragment unit in the target file that has not been uploaded is identified; at least one target fragment unit is taken as the upload object, and the process returns to the step of determining the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction.

[0153] It should be noted that the detailed functions of each module in the embodiments of this application can be found in the corresponding disclosure of the above-mentioned file upload method embodiments, and will not be repeated here.

[0154] This application also provides an electronic device in its embodiments. See also... Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device in this embodiment may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0155] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. When the electronic device is powered on, the RAM 503 also stores various programs and data required for the operation of the electronic device. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0156] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, memory cards, hard drives, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0157] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the file upload methods provided in this application.

[0158] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the file upload methods provided in this application.

[0159] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0161] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0162] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A file upload method, characterized in that, The file upload method includes: Obtain the target file, which is a file to be uploaded that meets the large file filtering criteria; In response to the upload operation of the target file, the target file is divided into multiple consecutive fragment units; The plurality of fragment units are used as upload objects. The upload objects are determined to be a first fragment unit to be uploaded in a first direction and a second fragment unit to be uploaded in a second direction. The first direction and the second direction are opposite directions. The first fragment unit is used for data transmission and the second fragment unit is used for real-time network speed detection. Based on the historical network speed measured at the previous time, a first sub-segment of the corresponding segment size is divided from the first segment unit and uploaded. At the same time, a second sub-segment of the specified segment size is divided from the second segment unit and uploaded. The current network speed at the current time is measured based on the upload result of the second sub-segment. The current network speed is used for the division and upload of sub-segments in the first segment unit at the next time. The second sub-segment of the specified segment size is used to avoid interference with the speed measurement result due to changes in the segment size.

2. The file upload method according to claim 1, characterized in that, The uploading of the first sub-fragment, which is divided into corresponding fragment sizes from the first fragment unit based on the historical network speed measured at the previous time, includes: Obtain the shard size configuration, which includes the maximum shard size and the minimum shard size; The segment size threshold for the slow start phase is determined based on the historical network speed, the preset security factor, and the preset ideal upload time. Obtain the size of the third sub-fragment that was most recently successfully uploaded in the first direction, and compare the size of the third sub-fragment with the fragment size threshold; If the size of the third sub-segment is less than the segment size threshold, the size of the third sub-segment is exponentially increased to obtain the candidate segment size of the first sub-segment; If the size of the third sub-segment is greater than or equal to the segment size threshold, the segment size of the third sub-segment is linearly increased according to the minimum segment size to obtain the candidate segment size of the first sub-segment; The dynamic fragment size of the first sub-fragment is determined based on the historical network speed and the ideal upload time; The minimum value among the maximum fragment size, the candidate fragment size, and the dynamic fragment size is used as the target fragment size of the first sub-fragment, and the target fragment size is used as the basis for the division and uploading of the first sub-fragment.

3. The file upload method according to claim 2, characterized in that, The step of uploading the first sub-fragment, which is divided into corresponding fragment sizes from the first fragment unit based on the historical network speed measured at the previous time, also includes: If the first sub-part fails to upload, the minimum sub-part size is taken as the target sub-part size, and the cumulative number of uploads of the first sub-part is counted. If the cumulative upload count does not reach the preset maximum upload count, wait to upload the first sub-part until other first part units in the first direction are successfully uploaded, then re-upload the first sub-part, and increment the cumulative upload count by 1. If the cumulative number of uploads reaches the preset maximum number of uploads, the first segment unit is determined to have failed to upload.

4. The file upload method according to claim 1, characterized in that, The step of measuring the current network speed at the current time based on the upload result of the second sub-segment includes: Get the upload time of the second sub-part; The current instantaneous network speed at the current time is determined based on the specified fragment size and the upload time. The current instantaneous network speed is stored as a data point in a sliding window, and the current smooth network speed is calculated after outlier removal from all data points in the sliding window.

5. The file upload method according to claim 4, characterized in that, The step of measuring the current network speed at the current time based on the upload result of the second sub-segment also includes: Obtain the historical average network speed, which is the average of the historical instantaneous network speed over a historical period; The current converged network speed is calculated based on the historical average network speed and the current smoothed network speed.

6. The file upload method according to claim 1, characterized in that, The upload task of the first segment unit is the first upload task, and the upload task of the second segment unit is the second upload task. The file upload method further includes: Initialize a thread pool, and create multiple concurrent threads in the thread pool. Among the multiple threads, the first thread is used to execute the first upload task, and the second thread is used to execute the second upload task. There is at least one first thread and one second thread, and the first thread corresponds one-to-one with the first upload task. The first thread is invoked to execute the first upload task, so as to divide the first sub-segment from the first segmentation unit and upload it; and the second thread is invoked to execute the corresponding second upload task, so as to divide the second sub-segment from the second segmentation unit and upload it, and measure the current network speed. Monitor the task execution status of the first thread; If the first sub-fragment under the first target thread in the first thread is successfully uploaded, a new first thread is created in the thread pool. If the first sub-fragment under the second target thread in the first thread is waiting to be uploaded, stop running the second target thread until the first fragment unit under the third target thread in the first thread is successfully uploaded, then restart the second target thread. If the first fragment unit under the fourth target thread in the first thread fails to upload, stop running the fourth target thread and add the fourth target thread to the failure queue; If the first shard unit under the fifth target thread in the first thread is successfully uploaded, stop running the fifth target thread and release the resources of the fifth target thread.

7. The file upload method according to claim 1, characterized in that, The file upload method further includes: In response to the breakpoint resume operation of the target file, at least one target fragment unit in the target file that has not been uploaded is identified; The at least one target fragment unit is taken as the upload object, and the process returns to the step of determining the first fragment unit to be uploaded in the first direction and the second fragment unit to be uploaded in the second direction.

8. A file upload device, characterized in that, The file upload device includes: The file acquisition module is used to acquire the target file, which is a file to be uploaded that meets the large file filtering criteria; The file upload module is used to respond to the upload operation of the target file by dividing the target file into multiple consecutive fragment units; using the multiple fragment units as upload objects, determining the first fragment unit to be uploaded in a first direction and the second fragment unit to be uploaded in a second direction, wherein the first direction and the second direction are opposite directions; the first fragment unit is used for data transmission, and the second fragment unit is used for real-time network speed detection; based on the historical network speed measured at the previous time, the module divides the first fragment unit into a first sub-fragment of a corresponding fragment size for upload; simultaneously, it divides the second sub-fragment of a specified fragment size from the second fragment unit for upload; and measures the current network speed at the current time based on the upload result of the second sub-fragment. The current network speed is used for the division and upload of sub-fragments in the first fragment unit at the next time; the second sub-fragment of a specified fragment size is used to avoid interference with the speed measurement result due to changes in fragment size.

9. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the file upload method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the file upload method as described in any one of claims 1 to 7.

Images