Dynamic fragment uploading method and device for large file and medium

Abstract

The invention discloses a dynamic fragment uploading method and device for a large file and a medium, and the method comprises the steps: monitoring the performance index of a client in real time to obtain a plurality of performance index data, dynamically adjusting the size of a fragment based on the performance index data, and dynamically adjusting the number of concurrent threads based on the performance index data; monitoring a server load state to obtain load state data, calculating a load residual value of the server nodes based on the load state data, and preferentially distributing a file fragment uploading task to the server node with the lowest load in all the server nodes by adopting a dynamic task distribution algorithm based on the load residual value; establishing a plurality of parallel connections between the client and the server based on a multiplexing protocol, uploading a plurality of file fragments in parallel, and recording fragment uploading progress through a breakpoint resume mechanism; and performing streaming verification and sequential merging on the successfully uploaded file fragments at the server side to realize incremental file reconstruction, and cleaning the temporary cache after merging is completed.

Description

Technical Field

[0001] This application relates to the field of large file technology, and in particular to a method, device and medium for dynamic fragmented uploading of large files. Background Technology

[0002] With the widespread application of cloud computing and mobile internet technologies, large file uploads have become an indispensable part of daily business operations. Especially in scenarios such as distributed storage, multimedia processing, and remote collaboration, upload efficiency and system stability directly affect user experience and operational costs. Traditional large file upload technologies generally adopt static fragmentation mechanisms, which transmit data through predefined fixed-size file fragments and the number of concurrent threads. This approach can maintain basic functionality under stable network environments and uniform load conditions, but its inherent defects are becoming increasingly apparent in real-world complex and variable network topologies and heterogeneous hardware platforms.

[0003] Existing static fragmented upload methods lack real-time awareness of client runtime status, making it impossible to dynamically adjust fragment size and concurrency strategies based on network bandwidth fluctuations, CPU utilization changes, and memory usage. Fixed fragment sizes are prone to transmission timeouts and fragment loss when network quality deteriorates, while in high-bandwidth environments, excessively small fragments generate excessive request overhead, resulting in low resource utilization. Furthermore, static concurrency control ignores the real-time load on client computing resources, easily leading to thread blocking or even application crashes in high CPU or memory usage scenarios, severely restricting the continuity and reliability of upload tasks.

[0004] At the server-side processing level, traditional solutions fail to effectively integrate the load status of distributed nodes. Task allocation is often based on simple round-robin or random strategies, lacking dynamic assessment of node processing capabilities. This leads to a continuous backlog of tasks on high-load nodes while low-load nodes remain idle, limiting the overall system throughput and making it prone to single-point overload failures during large-scale concurrent uploads. Furthermore, existing breakpoint resume mechanisms mostly rely on retransmitting the complete fragment list, failing to accurately record and restore partial upload states, which not only increases redundant data transmission but also exacerbates the storage and computing pressure on the server. Summary of the Invention

[0005] This application provides a method, device, and medium for dynamically uploading large files in chunks, in order to solve the above-mentioned technical problems.

[0006] On the one hand, embodiments of this application provide a method for dynamically segmented uploading of large files, including: The system monitors client performance metrics in real time to obtain multiple performance metrics data, and dynamically adjusts the shard size and the number of concurrent threads based on the performance metrics data; the performance metrics data include network bandwidth, CPU utilization, memory usage, and storage space capacity. Monitor the server load status to obtain load status data. Based on the load status data, calculate the remaining load value of the server node. Based on the remaining load value, use a dynamic task allocation algorithm to preferentially allocate file chunk upload tasks to the server node with the lowest load among all server nodes. Multiple parallel connections are established between the client and the server based on the multiplexing protocol to upload multiple file fragments in parallel, and the upload progress of the fragments is recorded through the breakpoint resume mechanism. On the server side, successfully uploaded file fragments are stream-based verified and sequentially merged to achieve incremental file reconstruction, and temporary cache is cleared after merging.

[0007] In one implementation of this application, client performance metrics are monitored in real time to obtain multiple performance metric data, specifically including: Based on the TCP / IP protocol layer, test data packets are actively sent and round-trip time and throughput are measured to obtain network bandwidth data. The processor usage is periodically queried through the operating system kernel interface to obtain CPU utilization data. The memory management module is invoked to detect the memory allocation status of the current application and system processes to obtain memory usage data, and the local file system directory structure is scanned to calculate the available storage capacity to obtain storage space data. The network bandwidth data, CPU utilization data, memory usage data, and storage space data are integrated to generate a comprehensive performance index dataset.

[0008] In one implementation of this application, dynamically adjusting the fragment size based on the performance index data specifically includes: The actual transmission time of the current fragment is compared with the preset ideal transmission time to calculate the transmission efficiency ratio. Based on the transmission efficiency ratio and network bandwidth, the fragment size is adjusted using a linear interpolation method. Specifically, the fragment size is increased when the network bandwidth increases and decreased when the network bandwidth decreases. Based on CPU utilization and memory usage, the fragment size is adjusted a second time. According to the new fragment size, the fragment sequence of the files to be uploaded is re-divided, and the fragment metadata information is updated.

[0009] In one implementation of this application, dynamically adjusting the number of concurrent threads based on the performance metric data specifically includes: Analyze CPU utilization, calculate remaining CPU capacity data by percentage conversion, and analyze memory usage to determine whether memory usage exceeds a preset safety threshold, thereby generating corresponding memory status flags; Based on CPU remaining capacity data and memory status flags, the initial number of concurrent threads is calculated using a weighted combination formula, and the initial number of concurrent threads is scaled and adjusted according to network bandwidth. The adjusted concurrent thread count data is added to the upload task queue to dynamically control the number of concurrent upload threads.

[0010] In one implementation of this application, monitoring the server load status to obtain load status data specifically includes: CPU utilization is periodically collected through the performance counter interface of the server-side operating system, and memory usage is obtained by calling the system memory management interface. Monitor disk read / write operation rates, collect disk I / O data through the I / O statistics module, and measure the input / output traffic of the server network interface to obtain network bandwidth data; The CPU utilization data, memory usage data, disk I / O data, and network bandwidth data are integrated to form a load status dataset.

[0011] In one implementation of this application, based on the load status data, the remaining load value of the server node is calculated, and based on the remaining load value, a dynamic task allocation algorithm is used to preferentially allocate file chunk upload tasks to the server node with the lowest load among all server nodes, specifically including: The CPU utilization and memory usage in the load status data are normalized to generate standardized load metrics. Determine the preset weighting coefficients corresponding to the CPU utilization and memory usage, calculate the remaining load value of each server node by weighted summation, and sort all available server nodes according to the remaining load value to generate a node priority queue. Based on the node priority queue, a round-robin algorithm is used to dynamically allocate file fragment upload tasks to the corresponding server nodes, and the load changes of the server nodes are monitored in real time to dynamically update the task allocation strategy.

[0012] In one implementation of this application, multiple parallel connections are established between the client and the server based on a multiplexing protocol to upload multiple file fragments in parallel, and the upload progress of each fragment is recorded through a breakpoint resume mechanism, specifically including: The client initializes an HTTP / 2 session, establishes multiple parallel data connection channels, and encapsulates file fragment data into HTTP requests so that they can be uploaded to the server simultaneously through multiplexed channels. The upload status data of each segment is recorded in the local storage on the client side, and upload interruption events are monitored in real time; the upload status data includes the uploaded segment identifier and progress information; If an upload interruption event is detected, the upload of incomplete segments can be automatically resumed by querying the upload status data stored locally, and the integrity of the segment upload can be verified and the upload status data can be updated.

[0013] In one implementation of this application, the server performs streaming verification and sequential merging on successfully uploaded file fragments to achieve incremental file reconstruction, and clears the temporary cache after merging. Specifically, this includes: For each received file fragment, a hash algorithm is applied to calculate a checksum, which is then compared with the expected value to verify integrity. The verified fragments are sorted according to the fragment number data to generate an ordered fragment sequence data, and the ordered fragment sequence data is read in a streaming manner and gradually merged into temporary file data; During the merging process, the file structure consistency is checked in real time to ensure the integrity of the merged file data. After the merging is completed, temporary fragment data is automatically deleted to release the occupied storage resources.

[0014] On the other hand, embodiments of this application also provide a dynamic chunking upload device for large files, the device comprising: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a dynamic chunking upload method for large files as described above.

[0015] On the other hand, this application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, implement the dynamic segmented upload method for large files as described above.

[0016] This application provides a method, device, and medium for dynamic fragmented uploading of large files, which has at least the following beneficial effects: By monitoring client network bandwidth, CPU utilization, and memory usage in real time, and dynamically adjusting fragment size based on transmission efficiency, the upload process can adapt to network fluctuations and device performance changes. When network quality deteriorates, fragment size is automatically reduced to lower the transmission failure rate; when network conditions are good, fragment size is appropriately increased to reduce request overhead. A mechanism for dynamically adjusting the number of concurrent threads based on client performance metrics effectively balances upload speed and system stability. By comprehensively analyzing CPU remaining capacity and memory usage, the number of simultaneously active upload threads is intelligently controlled, fully utilizing client processing power while avoiding system lag or crashes due to resource overload. By periodically collecting server node load status data, calculating remaining load values, and employing a dynamic task allocation algorithm, true distributed load balancing is achieved. This accurately assesses the processing capacity of each node and prioritizes upload tasks for nodes with lower loads, avoiding single-point overload. Combined with the parallel upload mechanism of the HTTP / 2 multiplexing protocol and robust breakpoint resume functionality, the efficiency and reliability of the transmission process are ensured. By establishing multiple parallel connections to upload fragments simultaneously and recording the upload progress locally on the client, the upload of incomplete fragments can be accurately recovered after a network interruption. The incremental file reconstruction method using streaming verification and sequential merging optimizes the server-side processing flow while ensuring file integrity. By independently verifying each fragment and merging them in numerical order, the server-side memory usage is reduced, processing efficiency is improved, and temporary data is automatically cleaned up after merging, effectively releasing storage resources. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a dynamic file chunking upload method for large files, provided as an embodiment of this application; Figure 2 This is a schematic diagram of the internal structure of a dynamic file chunking upload device for large files, provided as an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] Figure 1 This is a flowchart illustrating a dynamic file chunking upload method for large files, provided as an embodiment of this application.

[0021] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0022] It should be noted that the server can be a single device or a system composed of multiple devices, i.e., a distributed server. This application does not make any specific limitations on this.

[0023] like Figure 1 As shown in the embodiment of this application, a method for dynamic chunking and uploading of large files includes: Step 101: Monitor client performance metrics in real time to obtain multiple performance metric data, and dynamically adjust the shard size and the number of concurrent threads based on the performance metric data.

[0024] It should be noted that the performance metrics data in this application embodiment include network bandwidth, CPU utilization, memory usage, and storage space capacity.

[0025] In this embodiment, the acquisition of performance metrics data involves monitoring client status across multiple dimensions. This data serves as the foundational input for subsequent dynamic adjustment strategies. Specifically, performance metrics data includes network bandwidth, CPU utilization, memory usage, and storage capacity. These parameters collectively reflect the real-time operating status of the client. For example, network bandwidth data is obtained by actively sending test data packets at the TCP / IP protocol layer and measuring round-trip time and throughput. This method accurately captures fluctuations in the current network environment. CPU utilization data is obtained by periodically querying processor usage status through the operating system kernel interface, thereby understanding the load on computing resources. Memory usage data is collected by the memory management module detecting the memory allocation status of applications and system processes, ensuring timely identification of memory pressure. Storage capacity data is obtained by scanning the local file system directory structure and calculating available capacity to avoid upload interruptions due to insufficient storage. It is understood that the integration of these performance metrics data forms a comprehensive dataset.

[0026] After acquiring performance metrics data, the system dynamically adjusts the fragment size based on this data. Specifically, fragment size adjustment relies on transmission efficiency assessment, where the comparison between actual and ideal transmission time is used to calculate the transmission efficiency ratio. For example, when network bandwidth increases, the system appropriately increases the fragment size to reduce the number of requests and protocol overhead; conversely, when network bandwidth decreases or fluctuates significantly, the system decreases the fragment size to reduce the risk of transmission failure. Furthermore, fragment size adjustment also incorporates secondary corrections based on CPU utilization and memory usage. For instance, when CPU load is high, the system limits the increase in fragment size to prevent increased resource contention due to processing large fragments; when memory usage approaches a threshold, the system prioritizes reducing fragment size to control memory usage. The adjusted fragment size data is used to re-divide the fragment sequence of files to be uploaded and update fragment metadata information to ensure that subsequent upload tasks use the optimized fragment size.

[0027] Meanwhile, the system dynamically adjusts the number of concurrent threads based on performance metrics data. Understandably, controlling the number of concurrent threads aims to balance upload efficiency and system stability. Specifically, the system first parses CPU utilization data, converts it into a percentage to calculate remaining CPU capacity, reflecting the available computing resources on the client side; then it analyzes memory usage to determine if it exceeds a safe threshold and generates corresponding memory status flags. For example, when memory usage is low, the system allows more concurrent threads; while when memory usage is high, the system limits the number of concurrent threads to prevent memory overflow. Based on the remaining CPU capacity data and memory status flags, the system calculates the initial number of concurrent threads using a weighted combination formula and scales this initial value according to network bandwidth data to ensure that the number of threads does not exceed network capacity. Finally, the adjusted number of concurrent threads is applied to the upload task queue, dynamically controlling the number of simultaneously active upload threads, thereby achieving the optimal balance between resource utilization and system stability.

[0028] Step 102: Monitor the server load status to obtain load status data. Based on the load status data, calculate the remaining load value of the server node. Based on the remaining load value, use a dynamic task allocation algorithm to preferentially allocate the file chunk upload task to the server node with the lowest load among all server nodes.

[0029] In this embodiment, it should be noted that the load status data includes the server node's CPU utilization, memory usage, disk I / O data, and network bandwidth data. These metrics collectively describe the node's real-time processing capability. Specifically, CPU utilization data is periodically collected through the server operating system's performance counter interface, reflecting the node's computational load; memory usage data is obtained by calling the system memory management interface and is used to assess the node's memory usage; disk I / O data is collected by monitoring disk read / write operation rates and using the I / O statistics module, indicating the performance of the storage subsystem; and network bandwidth data is obtained by measuring the input / output traffic of the server's network interface, reflecting the node's network resource status. For example, these data are aggregated to form a load status dataset, providing a foundation for subsequent load assessment.

[0030] After acquiring load status data, the system calculates the remaining load value of the server nodes based on this data. Understandably, the remaining load value quantifies the available processing capacity of a node, and its calculation involves normalization and weighted summation of the load status data. Specifically, the system first normalizes CPU utilization and memory usage data to generate standardized load metrics, eliminating the influence of different metric dimensions. Then, it applies preset weight coefficients to these standardized load metrics and calculates the remaining load value for each server node through weighted summation. For example, the weight coefficients can be dynamically adjusted according to business needs; for instance, in compute-intensive tasks, CPU utilization may have a higher weight, while in memory-intensive tasks, memory usage may have a greater weight. A higher remaining load value indicates a lower node load and makes it more suitable for receiving new tasks.

[0031] Based on the calculated remaining load value, the system employs a dynamic task allocation algorithm to prioritize file chunk upload tasks to the server nodes with the lowest load. Specifically, the system sorts all available server nodes according to the remaining load value data, generating a node priority list or queue. For example, during task allocation, the system combines a round-robin algorithm with node priority to dynamically allocate file chunk upload tasks to low-load nodes. For instance, when the remaining load value of a node is significantly higher than other nodes, the system will prioritize allocating multiple chunk tasks to that node to fully utilize its processing capacity. Simultaneously, the system monitors server node load changes in real time and dynamically updates the task allocation strategy to ensure the continuous effectiveness of load balancing. This mechanism avoids the single point of overload problem that may occur in traditional solutions, improving the throughput and reliability of the entire distributed system.

[0032] Step 103: Establish multiple parallel connections between the client and the server based on the multiplexing protocol, upload multiple file fragments in parallel, and record the fragment upload progress through the breakpoint resume mechanism.

[0033] In this embodiment, the implementation of parallel upload and breakpoint resume mechanisms relies on modern network protocols and local storage technologies. It should be noted that the use of multiplexing protocols aims to improve transmission efficiency and reduce connection establishment overhead. Specifically, the system initializes an HTTP / 2 session on the client, establishing multiple parallel connection data channels. These channels allow multiple requests and responses to be transmitted simultaneously over a single TCP connection. For example, each file fragment is encapsulated as an independent HTTP request and uploaded to the server simultaneously through these multiplexed channels, thereby fully utilizing available network bandwidth and shortening the overall upload time. It is understood that this parallel upload method is particularly suitable for large file transfers because it can distribute file fragments across multiple channels, avoiding the bottleneck effect of a single channel.

[0034] During parallel uploads, the system records the upload progress of each segment using a breakpoint resume mechanism. Specifically, the client records the upload status data for each segment in local storage, including the uploaded segment identifier and progress information. For example, the upload status data may be stored in a structured format (such as JSON or database records), covering metadata such as segment number, upload timestamp, and checksum. The system monitors the upload process in real time, and automatically triggers a recovery mechanism when an upload interruption event is detected (such as network connection loss or client malfunction). Understandably, the recovery process queries the locally stored upload status data to accurately identify incomplete segments and retransmits only these segments, not the entire file. Furthermore, before resuming the upload, the system verifies the integrity of the segment uploads, for example, by comparing the confirmation information returned by the server to ensure data consistency and updating the upload status data to reflect the latest progress. This mechanism significantly improves upload reliability and user experience, especially in environments with unstable networks.

[0035] Step 104: On the server side, perform streaming verification and sequential merging on the successfully uploaded file fragments to achieve incremental file reconstruction, and clear the temporary cache after the merging is completed.

[0036] In this embodiment, the server-side processing and merging of file fragments is a crucial step in ensuring data integrity and system efficiency. It should be noted that streaming verification and sequential merging employ an incremental processing approach to reduce resource consumption and improve processing speed. Specifically, the system applies a hash algorithm (such as MD5 or SHA series) to each received file fragment to calculate a checksum and compares it with the expected value provided by the client to verify integrity. For example, if verification fails, the system requests the client to retransmit the fragment; if verification passes, the fragment is marked as available and enters the merging queue. It is understood that this streaming verification allows for real-time verification of fragments during the upload process, rather than waiting for all fragments to arrive before batch processing, thereby enabling early detection and correction of errors.

[0037] After the fragment verification passes, the system sorts the fragments according to their fragment numbers, generating an ordered fragment sequence data, and performs sequential merging to achieve incremental file reconstruction. Specifically, the system streams the ordered fragment sequence data and gradually writes the fragment data to temporary files or appends it directly to the target file. For example, during the merging process, the system checks the file structure consistency in real time to ensure that each fragment is correctly placed in its corresponding position within the file, preventing file corruption due to out-of-order errors. Understandably, this incremental merging method reduces server-side memory usage because it does not need to load all fragments into memory simultaneously, making it particularly suitable for handling very large files. After the merge is complete, the system automatically deletes temporary fragment data and cached files, releasing occupied storage resources. For example, the cleanup process may include deleting temporary fragment directories and updating file system indexes to ensure efficient use of storage space. Overall, this step, through an efficient verification and merging mechanism, ensures the final integrity of the file upload and system performance.

[0038] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a dynamic chunking upload device for large files, the structure of which is as follows: Figure 2 As shown.

[0039] Figure 2 This is a schematic diagram of the internal structure of a dynamic file chunking upload device for large files, provided as an embodiment of this application. Figure 2 As shown, the device includes: At least one processor; And, a memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, and the instructions, when executed by at least one processor, enable at least one processor to: Real-time monitoring of client performance metrics to obtain multiple performance metrics data, and dynamic adjustment of shard size and concurrent thread count based on performance metrics data; performance metrics data include network bandwidth, CPU utilization, memory usage, and storage space capacity; Monitor the server load status to obtain load status data. Based on the load status data, calculate the remaining load value of the server nodes. Based on the remaining load value, use a dynamic task allocation algorithm to preferentially allocate file chunk upload tasks to the server nodes with the lowest load among all server nodes. Multiple parallel connections are established between the client and the server based on the multiplexing protocol to upload multiple file fragments in parallel, and the upload progress of the fragments is recorded through the breakpoint resume mechanism. On the server side, successfully uploaded file fragments are stream-based verified and sequentially merged to achieve incremental file reconstruction, and temporary cache is cleared after merging.

[0040] This application also provides a non-volatile computer storage medium storing computer-executable instructions, which, when executed, can: Real-time monitoring of client performance metrics to obtain multiple performance metrics data, and dynamic adjustment of shard size and concurrent thread count based on performance metrics data; performance metrics data include network bandwidth, CPU utilization, memory usage, and storage space capacity; Monitor the server load status to obtain load status data. Based on the load status data, calculate the remaining load value of the server nodes. Based on the remaining load value, use a dynamic task allocation algorithm to preferentially allocate file chunk upload tasks to the server nodes with the lowest load among all server nodes. Multiple parallel connections are established between the client and the server based on the multiplexing protocol to upload multiple file fragments in parallel, and the upload progress of the fragments is recorded through the breakpoint resume mechanism. On the server side, successfully uploaded file fragments are stream-based verified and sequentially merged to achieve incremental file reconstruction, and temporary cache is cleared after merging.

[0041] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0042] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0043] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0048] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0049] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0050] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0051] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for dynamically segmenting and uploading large files, characterized in that, The method includes: The system monitors client performance metrics in real time to obtain multiple performance metrics data, and dynamically adjusts the shard size and the number of concurrent threads based on the performance metrics data; the performance metrics data include network bandwidth, CPU utilization, memory usage, and storage space capacity. Monitor the server load status to obtain load status data. Based on the load status data, calculate the remaining load value of the server node. Based on the remaining load value, use a dynamic task allocation algorithm to preferentially allocate file chunk upload tasks to the server node with the lowest load among all server nodes. Multiple parallel connections are established between the client and the server based on the multiplexing protocol to upload multiple file fragments in parallel, and the upload progress of the fragments is recorded through the breakpoint resume mechanism. On the server side, successfully uploaded file fragments are stream-based verified and sequentially merged to achieve incremental file reconstruction, and temporary cache is cleared after merging.

2. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Real-time monitoring of client performance metrics to obtain multiple performance metrics data, specifically including: Based on the TCP / IP protocol layer, test data packets are actively sent and round-trip time and throughput are measured to obtain network bandwidth data. The processor usage is periodically queried through the operating system kernel interface to obtain CPU utilization data. The memory management module is invoked to detect the memory allocation status of the current application and system processes to obtain memory usage data, and the local file system directory structure is scanned to calculate the available storage capacity to obtain storage space data. The network bandwidth data, CPU utilization data, memory usage data, and storage space data are integrated to generate a comprehensive performance index dataset.

3. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Dynamically adjusting the fragment size based on the aforementioned performance metrics data specifically includes: The actual transmission time of the current fragment is compared with the preset ideal transmission time to calculate the transmission efficiency ratio. Based on the transmission efficiency ratio and network bandwidth, the fragment size is adjusted using a linear interpolation method. Specifically, the fragment size is increased when the network bandwidth increases and decreased when the network bandwidth decreases. Based on CPU utilization and memory usage, the fragment size is adjusted a second time. According to the new fragment size, the fragment sequence of the files to be uploaded is re-divided, and the fragment metadata information is updated.

4. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Dynamically adjusting the number of concurrent threads based on the aforementioned performance metrics data specifically includes: Analyze CPU utilization, calculate remaining CPU capacity data by percentage conversion, and analyze memory usage to determine whether memory usage exceeds a preset safety threshold, thereby generating corresponding memory status flags; Based on CPU remaining capacity data and memory status flags, the initial number of concurrent threads is calculated using a weighted combination formula, and the initial number of concurrent threads is scaled and adjusted according to network bandwidth. The adjusted concurrent thread count data is added to the upload task queue to dynamically control the number of concurrent upload threads.

5. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Monitor server load status to obtain load status data, specifically including: CPU utilization is periodically collected through the performance counter interface of the server-side operating system, and memory usage is obtained by calling the system memory management interface. Monitor disk read / write operation rates, collect disk I / O data through the I / O statistics module, and measure the input / output traffic of the server network interface to obtain network bandwidth data; The CPU utilization data, memory usage data, disk I / O data, and network bandwidth data are integrated to form a load status dataset.

6. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Based on the load status data, the remaining load value of the server nodes is calculated. Then, based on this remaining load value, a dynamic task allocation algorithm is used to preferentially allocate file chunk upload tasks to the server nodes with the lowest load among all server nodes. Specifically, this includes: The CPU utilization and memory usage in the load status data are normalized to generate standardized load metrics. Determine the preset weighting coefficients corresponding to the CPU utilization and memory usage, calculate the remaining load value of each server node by weighted summation, and sort all available server nodes according to the remaining load value to generate a node priority queue. Based on the node priority queue, a round-robin algorithm is used to dynamically allocate file fragment upload tasks to the corresponding server nodes, and the load changes of the server nodes are monitored in real time to dynamically update the task allocation strategy.

7. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, Multiple parallel connections are established between the client and server based on a multiplexing protocol to upload multiple file fragments in parallel. A breakpoint resumption mechanism is used to record the fragment upload progress, specifically including: The client initializes an HTTP / 2 session, establishes multiple parallel data connection channels, and encapsulates file fragment data into HTTP requests so that they can be uploaded to the server simultaneously through multiplexed channels. The upload status data of each segment is recorded in the local storage on the client side, and upload interruption events are monitored in real time; the upload status data includes the uploaded segment identifier and progress information; If an upload interruption event is detected, the upload of incomplete segments can be automatically resumed by querying the upload status data stored locally, and the integrity of the segment upload can be verified and the upload status data can be updated.

8. The method for dynamic chunking and uploading of large files according to claim 1, characterized in that, On the server side, successfully uploaded file fragments are subjected to streaming verification and sequential merging to achieve incremental file reconstruction. After merging, temporary cache is cleared. Specifically, this includes: For each received file fragment, a hash algorithm is applied to calculate a checksum, which is then compared with the expected value to verify integrity. The verified fragments are sorted according to the fragment number data to generate an ordered fragment sequence data, and the ordered fragment sequence data is read in a streaming manner and gradually merged into temporary file data; During the merging process, the file structure consistency is checked in real time to ensure the integrity of the merged file data. After the merging is completed, temporary fragment data is automatically deleted to release the occupied storage resources.

9. A dynamic file chunking upload device for large files, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a dynamic chunking upload method for large files as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, a dynamic chunking upload method for large files as described in any one of claims 1-8 is implemented.

Images