Efficient material downloading system and method based on multi-node dynamic concurrency control

Abstract

The application discloses a high-efficiency material downloading system and method based on multi-node dynamic concurrent control, relates to the technical field of resource scheduling, and comprises the following steps: establishing a source station row according to source station information and establishing a size column according to file size information in each downloading node, and setting a waiting task queue, a retry task queue and an execution slot in a downloading execution unit formed by the cross; writing a material task to be downloaded into a corresponding downloading execution unit; collecting task connection establishment, first packet waiting, failure retry, effective transmission and continuous transmission segment information, determining column dissipation difference, unit net continuous length and unit pressure release slot boundary; and adjusting the execution slot and migrating the task according to the information. Through the realization of task structural shunting, execution slot fine control and cross-node collaborative scheduling, the effective transmission proportion, node stability and overall downloading throughput are improved.

Description

Technical Field

[0001] This invention relates to the field of resource scheduling technology, specifically to a high-efficiency material download system and method based on multi-node dynamic concurrency control. Background Technology

[0002] With the rapid development of digital advertising, short video distribution, content moderation, and material analysis, the number of materials to be downloaded continues to grow. These materials primarily include video files, image files, and other multimedia resources. They are typically sourced from diverse locations, are numerous, and vary widely in size, ranging from small files of a few megabytes to large files of hundreds of megabytes. To meet the needs of subsequent identification, analysis, storage, and processing, it is usually necessary to download massive amounts of materials from different source sites to local or cloud nodes within a short period.

[0003] Existing material download solutions typically employ either a single-node multi-threaded download approach or a multi-node distributed concurrent download approach. Single-node solutions generally use a unified thread pool to concurrently load and execute download tasks; multi-node distributed solutions typically divide the download tasks and distribute them to multiple download nodes, each of which then downloads according to a preset concurrency level. Some existing solutions further adjust the thread pool size or the number of concurrent connections based on fundamental operational metrics such as processor utilization, memory utilization, bandwidth utilization, failure rate, or timeout rate to improve overall download efficiency.

[0004] However, in practical applications, the above solutions still have significant limitations. First, existing solutions typically place all tasks into the same waiting queue or thread pool for processing, lacking a structured distinction between tasks from different origin servers and tasks with different file sizes. This leads to small file tasks and large file tasks competing for execution under the same concurrent resources. Since small file tasks usually involve more frequent connection establishment, first packet waiting, and failure retries, while large file tasks tend to be more focused on continuous and effective transmission, the mixed execution of the two can easily result in significant differences in resource consumption characteristics that are difficult to regulate uniformly.

[0005] Secondly, existing concurrency control methods mostly rely on the resource status of the entire node, such as whether bandwidth has reached a threshold, whether processors have exceeded preset limits, and whether the failure rate has increased, and then adjust the number of download threads accordingly. While this type of control can achieve a certain degree of dynamic adjustment, its control object is still the unified thread pool of the entire node, making it difficult to accurately reflect the actual transmission status of specific task categories within the node. In particular, it is difficult to identify resource consumption caused by ineffective transmission behaviors such as connection establishment, first packet waiting, and failure retries, and it is also difficult to distinguish whether a certain type of task is truly suitable for further increasing concurrency.

[0006] Furthermore, in multi-node download scenarios, there are significant differences in the response capabilities, rate limiting strategies, and transmission characteristics of tasks with different file sizes across different source sites. Existing solutions lack a control mechanism that first finely organizes tasks within a node according to source site and file size, and then performs unit-level concurrency adjustment for specific execution units, resulting in overly coarse-grained concurrency control. When a certain type of task consistently exhibits high connection dissipation or low effective continuous transmission capacity in a specific node, existing solutions often can only reduce concurrency at the overall node level, making it difficult to specifically reduce or migrate problematic tasks, thus affecting overall download throughput and node operational stability. Summary of the Invention

[0007] Based on the shortcomings of the prior art described above, the purpose of this invention is to provide an efficient material download system and method based on multi-node dynamic concurrency control, so as to solve the above-mentioned technical problems.

[0008] To achieve the above objectives, the present invention provides the following technical solution: an efficient material download method based on multi-node dynamic concurrency control, comprising: Within each download node, a source station row is created based on the source station information, and a size column is created based on the file size information. Within the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue, and multiple execution slots are set up. Write the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Within the sampling window, the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task of the execution slot are collected. The column dissipation difference is determined based on the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, and effective transmission time for tasks within the same size column. The unit net continuous transmission length is determined based on the continuous transmission segment length of each execution slot within the same download execution unit. The unit pressure relief slot boundary is generated based on the column dissipation difference, the unit net continuous transmission length, the number of currently enabled slots, the failure rate, the timeout rate, and the remaining bandwidth capacity of the node. Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the difference between the column dissipation difference of the previous sampling window and the current sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the current sampling window and the net continuous length of the unit output of the previous sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are added. When the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the difference between the column dissipation difference of the current sampling window and the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the previous sampling window and the net continuous length of the unit output of the current sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are reduced. When the column dissipation difference of adjacent sampling windows increases window by window and the net continuous length of the unit output decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

[0009] The present invention is further configured to create source site rows based on source site information and size columns based on file size information, including: Create a corresponding source row based on at least one of the following: source domain name, source address, or source identifier of the material to be downloaded; Establish corresponding size columns based on the preset size range to which the file size of the material to be downloaded belongs; the size columns include at least small file columns, medium file columns, and large file columns. Downloadable materials with the same source site information and file sizes falling within the same preset size range will be grouped into the same download execution unit.

[0010] The present invention is further configured to write the material task to be downloaded into the corresponding download execution unit according to the source site information and file size information, including: Determine the corresponding source site row based on the source site information of the material task to be downloaded, and determine the corresponding size column based on the file size information of the material task to be downloaded; Write the task of downloading the materials to be downloaded into the waiting task queue of the determined download execution unit; The number of execution slots initially enabled is determined based on the initial capacity of the download node and the number of tasks in the corresponding download execution unit. The material tasks to be downloaded, corresponding to the number of execution slots initially enabled, are retrieved from the waiting task queue and loaded into the execution slots for download.

[0011] The present invention is further configured to collect, within the sampling window, the number of connection establishments, average connection establishment time, first packet wait time, number of failed retries, effective transmission duration, and length of continuous transmission segments for the task corresponding to the execution slot, including: Record the connection establishment start time, connection establishment completion time, first packet arrival time, entry into valid transmission state time, transmission interruption time, timeout occurrence time, and completion time for the task corresponding to the execution slot; The connection establishment time is determined based on the connection establishment start time and connection establishment completion time; the first packet waiting time is determined based on the connection establishment completion time and the first packet arrival time; and the continuous transmission segment length is determined based on the duration between the time of entering the effective transmission state and the time of transmission interruption, timeout occurrence, or completion. The number of connection establishments and failure retries for each task in each execution slot within the statistical sampling window are recorded.

[0012] The present invention is further configured to determine the column dissipation difference based on the number of connection establishments, average connection establishment time, first packet wait time, number of failed retries, and effective transmission time within the same size column, including: The average connection establishment time for tasks within the same size column within the sampling window is summed up by the number of connection establishments to obtain the total connection establishment time; The total first packet waiting time is obtained by summing up the first packet waiting times of tasks within the same size column within the sampling window. Multiply the number of failed retries for tasks within the same size column in the sampling window by the preset single retry duration to obtain the total retry duration; The effective transmission time of tasks within the same size column within the sampling window is summed to obtain the total effective transmission time. The column dissipation difference for this size column is obtained by dividing the sum of the total connection establishment time, the total first packet wait time, and the total retry time by the total effective transmission time.

[0013] The present invention is further configured to determine the net continuous transmission length of the unit based on the continuous transmission segment length of each execution slot within the same download execution unit, including: Extract the length of continuous transmission segments from the entry of a valid transmission state to the occurrence of transmission interruption, timeout, or completion within the sampling window for each execution slot in the same download execution unit; The total continuous transmission length is obtained by summing the lengths of the continuous transmission segments. Count the number of consecutive transmission segments; Divide the total continuous transmission length by the number of continuous transmission segments to obtain the net continuous transmission length of the download execution unit.

[0014] The present invention is further configured to generate cell stress relief slot boundaries based on column dissipation difference, net cell output continuity, number of currently active slots, failure rate, timeout rate, and remaining node bandwidth capacity, including: The initial slot boundaries are defined by the number of currently active slots. When the column dissipation difference is less than the preset upper limit of dissipation difference, the net continuous length of the unit is greater than the preset lower limit of continuous length, the failure rate and timeout rate are both lower than the corresponding preset thresholds, and the remaining bandwidth capacity of the node is higher than the preset bandwidth threshold, the initial slot boundary is increased by the preset slot boundary adjustment value to obtain the unit pressure relief slot boundary. When the column dissipation difference is greater than the preset upper limit of dissipation difference, the net continuous length of the unit is less than the preset lower limit of continuous length, the failure rate or timeout rate is higher than the corresponding preset threshold, or the remaining bandwidth capacity of the node is lower than the preset bandwidth threshold, the initial slot boundary is subtracted from the preset slot boundary adjustment value to obtain the unit pressure relief slot boundary.

[0015] The present invention is further configured to increase a preset number of execution slots, including: when the conditions for increasing execution slots are met, enabling a preset number of inactive execution slots; selecting tasks to be downloaded from the waiting task queue of the corresponding download execution unit and loading them into the newly added execution slots according to the task priority in the waiting task queue or the order in which they entered the waiting task queue; after loading tasks into the newly added execution slots, recalculating the column dissipation difference, the unit net output continuity length, and the unit pressure relief slot boundary in the next sampling window; Reducing the preset number of execution slots includes: stopping the loading of new downloadable material tasks into the preset number of execution slots in the corresponding download execution unit when the conditions for reducing execution slots are met; keeping the loaded tasks running until completion, or suspending tasks that support resume download and writing them into the retry task queue; and setting the corresponding execution slot to an inactive state after the loaded tasks are completed or suspended.

[0016] The present invention is further configured to migrate waiting tasks and retry tasks to corresponding download execution units on other download nodes, including: In three consecutive sampling windows, if the column dissipation difference of each subsequent sampling window minus the column dissipation difference of its preceding sampling window is greater than or equal to a preset dissipation difference threshold, and the net continuous length of the unit input of each preceding sampling window minus the net continuous length of the unit input of each subsequent sampling window is greater than or equal to a preset continuous length threshold, then the current download execution unit is determined to meet the migration conditions. Select waiting tasks and retry tasks as migration tasks from the download execution units that meet the migration conditions; Among other download nodes, select a download execution unit that has the same source station information and file size range as the migration task, and whose currently enabled number of slots is less than its unit release slot boundary, as the target download execution unit, and write the migration task into the target download execution unit's waiting task queue or retry task queue.

[0017] This invention also provides a high-efficiency material download system based on multi-node dynamic concurrency control, used to implement the above-mentioned high-efficiency material download method based on multi-node dynamic concurrency control, including: Structure building module: In each download node, a source station row is created according to the source station information and a size column is created according to the file size information. In the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue and multiple execution slots are set up. Task writing module: Writes the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Parameter generation module: Collects the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task in the sampling window. Determines the column dissipation difference based on the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, and effective transmission time for tasks within the same size column. Determines the unit net continuous transmission length based on the continuous transmission segment length of each execution slot in the same download execution unit. Generates the unit pressure relief slot boundary based on the column dissipation difference, unit net continuous transmission length, number of currently enabled slots, failure rate, timeout rate, and remaining bandwidth capacity of the node. Slot control module: Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the column dissipation difference of the previous sampling window minus the column dissipation difference of the current sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the current sampling window minus the unit net output continuity length of the previous sampling window is greater than or equal to a preset continuity length threshold, the number of execution slots is reduced by a preset number when the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the column dissipation difference of the current sampling window minus the column dissipation difference of the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the previous sampling window minus the unit net output continuity length of the current sampling window is greater than or equal to a preset continuity length threshold. When the column dissipation difference of adjacent sampling windows increases window by window and the unit net output continuity length decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

[0018] This invention provides a high-efficiency material download system and method based on multi-node dynamic concurrency control. The method establishes source station rows based on source station information and size columns based on file size information within each download node. Within the download execution unit formed by the intersection of the source station rows and size columns, a waiting task queue, a retry task queue, and multiple execution slots are set up. Material tasks to be downloaded are written into the corresponding download execution unit according to the source station information and file size information. Within a sampling window, the number of connection establishments, average connection establishment time, first packet waiting time, number of failed retries, effective transmission time, and continuous transmission segment length for the tasks corresponding to the execution slots are collected. The column dissipation difference is determined based on the number of connection establishments, average connection establishment time, first packet waiting time, number of failed retries, and effective transmission time for tasks within the same size column. The unit's net continuous transmission length is determined based on the continuous transmission segment length of each execution slot within the same download execution unit. Finally, the method considers the column dissipation difference, the unit's net continuous transmission length, the number of currently active slots, the failure rate, and the timeout rate. The remaining bandwidth capacity of the node generates the unit pressure relief slot boundary; starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the column dissipation difference of the previous sampling window minus the column dissipation difference of the current sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net input continuity length of the current sampling window minus the unit net input continuity length of the previous sampling window is greater than or equal to a preset continuity length threshold, the number of execution slots is increased by a preset number when the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the column dissipation difference of the current sampling window minus the column dissipation difference of the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net input continuity length of the previous sampling window minus the unit net input continuity length of the current sampling window is greater than or equal to a preset continuity length threshold, the number of execution slots is decreased by a preset number; when the column dissipation difference of adjacent sampling windows increases window by window and the unit net input continuity length decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes. The beneficial effects include: 1. By establishing source site rows based on source site information and size columns based on file size information within each download node, and setting up waiting task queues, retry task queues, and multiple execution slots within the download execution unit formed by the intersection of the source site rows and size columns, the download material tasks that originally entered the same thread pool are restructured into a structured set of tasks organized hierarchically according to source site category and file size range. This effectively reduces resource contention between high-frequency connection tasks for small files and continuous transmission tasks for large files, improves the orderliness of task organization within nodes and the clarity of execution boundaries, and enhances task diversion and structured management capabilities during the material download process. 2. By collecting data such as the number of connection establishments, average connection establishment time, first packet wait time, number of failed retries, effective transmission duration, and continuous transmission segment length within the sampling window, the column dissipation difference, unit net continuous transmission length, and unit pressure relief slot boundary are further constructed. These serve as the criteria for determining the increase, decrease, and maintenance of execution slots. This allows concurrency control to move beyond coarse-grained adjustments to the overall number of threads or connections of a node, enabling fine-grained control based on the actual transmission status of a specific download execution unit. It can more accurately identify the impact of ineffective transmission behaviors such as connection establishment, first packet wait, and failed retries on the download process, dynamically adjusting execution slots based on continuous effective transmission capacity and remaining bandwidth capacity. This increases the proportion of effective transmission, reduces invalid connection usage, and enhances the stable operation of download nodes in complex material task scenarios. 3. By detecting the changing trends of column dissipation difference and unit net transmission continuity within multiple consecutive sampling windows, when it is determined that the current download execution unit is continuously in a state of high dissipation and low continuity transmission, waiting tasks and retry tasks are migrated to the corresponding download execution units of other download nodes for execution. This allows problematic tasks to be transferred from nodes that are currently unsuitable for continued operation to nodes that are more suitable for execution. While maintaining the relatively stable operation of already loaded tasks, this allows for cross-node reallocation of subsequent tasks to be executed, avoiding long-term limitations on local nodes that lead to a decline in overall download efficiency. This is beneficial for improving the task scheduling flexibility, load balancing capabilities, and overall throughput performance of downloading massive amounts of material in a multi-node cluster environment.

[0019] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart illustrating an efficient material download method based on multi-node dynamic concurrency control, as an exemplary embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0023] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0024] Example 1: An efficient method for downloading media based on multi-node dynamic concurrency control, such as Figure 1 As shown, it includes: Within each download node, a source station row is created based on the source station information, and a size column is created based on the file size information. Within the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue, and multiple execution slots are set up. Write the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Within the sampling window, the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task of the execution slot are collected. The column dissipation difference is determined based on the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, and effective transmission time for tasks within the same size column. The unit net continuous transmission length is determined based on the continuous transmission segment length of each execution slot within the same download execution unit. The unit pressure relief slot boundary is generated based on the column dissipation difference, the unit net continuous transmission length, the number of currently enabled slots, the failure rate, the timeout rate, and the remaining bandwidth capacity of the node. Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the difference between the column dissipation difference of the previous sampling window and the current sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the current sampling window and the net continuous length of the unit output of the previous sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are added. When the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the difference between the column dissipation difference of the current sampling window and the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the previous sampling window and the net continuous length of the unit output of the current sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are reduced. When the column dissipation difference of adjacent sampling windows increases window by window and the net continuous length of the unit output decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

[0025] In this embodiment, after each download node starts, it first loads a preset execution structure configuration table. This table includes at least source site identification rules, file size classification rules, download execution unit mapping rules, waiting task queue configuration rules, retry task queue configuration rules, and execution slot quantity configuration rules. Upon receiving a task to download materials, the download node first extracts the material download link, source site domain name, source site address, source site identifier, file size, task priority, and task status flag from the task description information. For tasks lacking source site information, the source site domain name or address can be obtained by parsing the material download link, and the parsed information is written into the task metadata to ensure a unified data input source for subsequent source site establishment processes.

[0026] After standardizing the task information, the download node establishes source site rows based on the source site information. Specifically, any one of the source site domain name, source site address, or source site identifier can be used as the source site category key. Alternatively, the source site identifier can be selected first according to a preset priority, and then fall back to the source site domain name and source site address if the source site identifier is missing. Tasks of materials to be downloaded with the same category key are uniformly grouped into the same source site row. Each source site row corresponds to an independent task organization unit within the node, and the node maintains the row identifier, current number of tasks, current number of active slots, number of waiting tasks, number of retrying tasks, and statistics window cache corresponding to that source site row. After adopting the above method, tasks from the same source site are grouped into the same organizational level, which is beneficial for subsequent statistics on the connection establishment characteristics, failure retry characteristics, and continuous transmission characteristics of tasks corresponding to that source site, and avoids the distortion of statistical results caused by the mixing of tasks from different source sites.

[0027] After the origin server row is established, size columns are further created based on file size information. Specifically, multiple file size ranges are pre-defined. For example, tasks with file sizes smaller than a first size threshold are classified as small files, tasks with file sizes between the first and second size thresholds are classified as medium files, and tasks with file sizes larger than the second size threshold are classified as large files. For tasks to be downloaded, the corresponding size column is written according to the range in which the file size falls. To ensure applicability under different business scenarios, the first and second size thresholds can be configured based on historical material sample statistics or preset manually. The purpose of setting the size columns is to distinguish between small file tasks that are frequently established and sensitive to the first packet wait time and large file tasks that have a long continuous transmission time, so that subsequent control logic can further subdivide scheduling objects according to file size characteristics within the same origin server.

[0028] After the source row and size column are established, the node forms multiple download execution units based on the intersection of the source row and size column. Each download execution unit uniquely corresponds to one source row and one size column. That is, small file tasks, medium file tasks, and large file tasks in the same source are assigned to different download execution units, and tasks with the same size range in different source stations are also placed in different download execution units. Thus, the download tasks that originally entered the node thread pool uniformly are restructured into multiple task sets with clear boundaries. Each task set corresponds to subsequent independent statistics, adjustment, and migration objects. To facilitate operation control, each download execution unit establishes a unit identifier, and the node maintains a waiting task queue, a retry task queue, and an execution slot set corresponding to the unit identifier.

[0029] The waiting task queue stores tasks that have not yet been loaded into execution slots for download. The retry task queue stores tasks that time out, fail, or are suspended during execution and await rescheduling. Execution slots are independently operable download execution points, with each slot loading only one download task at a time. Execution slots can be implemented using arrays, linked lists, or slot tables. Nodes maintain information for each execution slot, including slot status, loaded task identifiers, loading start time, connection establishment status, valid transmission status, and completion status. Initially, nodes allocate an initial number of execution slots to each download execution unit based on the number of tasks within the current unit, the node's initial capacity, and preset slot allocation rules. The nodes then retrieve the corresponding number of tasks from the waiting task queue of that unit in a preset order and load them into the enabled execution slots for download.

[0030] In actual operation, if the source site information of a downloadable material task is the same advertising material source site, and the file size is eight megabytes, then the task is first written to the corresponding source site row, and then classified into the small file column according to the preset size range corresponding to eight megabytes. Finally, it enters the waiting task queue of the download execution unit formed by the intersection of the source site row and the small file column. If another task comes from the same source site but the file size is two hundred megabytes, it will enter another download execution unit corresponding to the large file column under the same source site row. It can be seen that even if two tasks come from the same source site, as long as the file size is different, they will be classified into different download execution units; similarly, even if two tasks have the same file size, as long as they come from different source sites, they will be classified into different download execution units. After adopting the above method, the node can perform execution slot control, statistical analysis, and migration determination for tasks in different download execution units, thereby improving the structured scheduling capability in mixed download scenarios of multiple types of materials.

[0031] To ensure the consistency of this structure during node operation, as new download tasks continuously arrive, the node repeatedly executes the processes of source station information parsing, source station row creation, size column determination, and download execution unit mapping. If the source station row and size column combination corresponding to the current task already exist, they are directly written into the existing download execution unit; if the source station row corresponding to the current task exists but the size column combination does not exist, a corresponding size column unit is created under that source station row; if the source station row corresponding to the current task does not exist, the source station row is created first, and then the corresponding size column and download execution unit are created according to the file size. Through the above method, the download node can dynamically maintain the mapping relationship between source station rows, size columns, and download execution units during operation, ensuring that download tasks are always organized in an orderly manner according to both source station and file size dimensions. This provides a unified and feasible structural foundation for subsequent calculations of column dissipation difference, unit net input continuity length, and unit pressure relief groove boundaries.

[0032] After establishing the source site row, size column, and download execution unit, the download node maps the execution units of the materials to be downloaded entering the node. Specifically, the node first reads the source site information and file size information carried in the materials to be downloaded, and determines the source site row and corresponding size column to which the task belongs based on the pre-established source site row mapping table and size column mapping table. If the task metadata contains a source site identifier, source site domain name, and source site address simultaneously, one of these pieces of information can be selected as the source site mapping basis for the current task according to a preset priority; if the source site identifier is missing, the mapping is completed using either the source site domain name or the source site address. Subsequently, the node retrieves the corresponding download execution unit identifier based on the determined source site row and size column combination, and establishes a binding relationship between the task and the download execution unit identifier to ensure that the same task subsequently enters the correct waiting task queue and execution slot control range.

[0033] After identifying the target download execution unit, the node writes the tasks to be downloaded into the waiting task queue of that unit. The waiting task queue can be organized by task arrival order, task priority, or a preset sorting strategy. To avoid queue chaos during subsequent loading, the node synchronously records the task's enqueue time, source row identifier, size column identifier, task priority, retry count, and current status flag when a task is written into the waiting task queue. For tasks entering the download execution unit for the first time, their status flag can be set to "pending loading." For tasks returning from the retry task queue to the waiting task queue, their retry count and latest status flag are updated while maintaining the original task identifier. In this way, the node ensures that tasks within the same download execution unit enter the subsequent loading process according to unified management rules.

[0034] After the tasks to be downloaded are written into the waiting task queue, the node determines the number of execution slots to be initially enabled based on the initial capacity of the download node and the number of tasks in the corresponding download execution unit. Specifically, the download node can preload initial capacity parameters at startup. These initial capacity parameters include at least the maximum total number of execution slots that the node can enable simultaneously, the maximum number of initial slots that a single download execution unit can enable, and the initial slot allocation rules corresponding to different size columns. When the number of waiting tasks in a download execution unit is small, the node only enables execution slots matching the number of waiting tasks; when the number of waiting tasks in a download execution unit is large, the node allocates the number of initially enabled execution slots to that download execution unit, provided that it does not exceed the unit's initial slot limit and the node's total execution slot limit. In this way, the initial loading process not only considers the number of tasks but also takes into account the overall capacity of the node and the local execution capacity of the unit.

[0035] After determining the initial number of execution slots to be enabled, the node retrieves the downloadable material tasks corresponding to the initial number of execution slots from the waiting task queue of the corresponding download execution unit, and loads them one by one into the execution slots for download. During the loading process, each execution slot generates a corresponding slot operation record after a task is loaded. The slot operation record includes at least the execution slot identifier, the currently loaded task identifier, the loading start time, the current connection status, and the current execution status. If the number of tasks in the waiting task queue is less than the initial number of execution slots to be enabled, the node only loads the existing tasks, and the remaining execution slots remain in an inactive state. If the number of tasks in the waiting task queue is greater than the initial number of execution slots to be enabled, the remaining tasks continue to be retained in the waiting task queue, waiting for subsequent addition of execution slots or completion of already loaded tasks before entering the loading process.

[0036] In a specific implementation scenario, if a download node currently has a total execution slot limit of thirty, and a small file column download execution unit corresponding to a certain source site row contains six material tasks to be downloaded, while the initial slot limit for this download execution unit is set to three, then the node will only activate three execution slots in this download execution unit and sequentially retrieve the first three material tasks to be downloaded from the waiting task queue for loading and execution, while the remaining three tasks remain in the waiting task queue. If a large file column download execution unit corresponding to the same source site row contains only one material task to be downloaded, then the node will only activate one execution slot in this download execution unit and load that task, avoiding empty slots occupying resources due to mechanically activating multiple execution slots. Thus, different download execution units can perform differentiated initial loading according to their respective task quantities and the node's initial capacity.

[0037] To ensure the orderly loading process, after a node retrieves a task from the waiting task queue and loads it into an execution slot, it can simultaneously update the number of tasks, the number of active slots, and the number of waiting tasks in the corresponding download execution unit. If an execution slot completes its current task or is released due to an anomaly, the node re-checks the waiting task queue of that download execution unit. If there are still tasks to be processed in the queue and the number of active slots does not exceed the subsequently calculated unit release slot limit, the next downloadable material task is selected from the waiting task queue and loaded into an idle execution slot. If the number of active slots has reached the control limit, the idle execution slot is kept from immediately loading new tasks. Through this processing method, downloadable material tasks are continuously written to, retained, and loaded into the corresponding download execution unit, forming a complete closed loop between task mapping, queue management, and execution slot activation.

[0038] For tasks originating from different source stations but with the same file size, even if the file sizes all fall into the same size column, they will be written into the waiting task queues of different download execution units because their source station rows are different. Conversely, for tasks originating from the same source station but with different file sizes, even if their source station rows are the same, they will enter different download execution units because their size columns are different. Therefore, the tasks to be downloaded are not simply written into a unified node queue, but are accurately written into the corresponding download execution unit according to the source station information and file size information, and the waiting, loading, and execution processes are completed within that unit. This writing method provides clear task ownership boundaries for subsequent calculations of column dissipation difference, unit net input continuity length, and unit pressure relief groove boundaries, thereby improving the structure and feasibility of the entire download control process.

[0039] After the downloadable materials are loaded into the execution slots of each download execution unit, the download nodes begin to continuously collect the running status of the tasks corresponding to the execution slots according to a preset sampling window. Specifically, the node establishes a slot operation record for each execution slot, continuously writing the connection establishment start time, connection establishment completion time, first packet arrival time, entry into effective transmission state time, transmission interruption time, timeout occurrence time, and completion time of the task corresponding to that execution slot into the slot operation record. The sampling window can be set according to a fixed time length, such as continuously advancing in second-level time slices, or it can form discrete sampling segments according to a preset number of samples. At the end of each sampling window, the node summarizes the operation records of all execution slots within the window to form the basic data required for subsequent column dissipation difference, unit net output continuity length, and unit pressure relief slot boundary.

[0040] Within the sampling window, the node first determines the connection establishment time for a single task based on the connection establishment start time and connection establishment completion time, and determines the first packet waiting time based on the connection establishment completion time and the first packet arrival time. If a task experiences multiple connection establishments or failed reconnections within the current sampling window, the connection establishment time and first packet waiting time for each instance are recorded separately and accumulated during window aggregation. For the number of failed retries, the node counts the number of times the task re-establishes a connection or reloads the execution slot within the current sampling window. For the effective transmission duration, the duration of the effective transmission state is measured from the time the task enters the effective transmission state to the time of transmission interruption, timeout occurrence, or completion. In this way, the node can obtain basic operational information such as the number of connection establishments, average connection establishment time, first packet waiting time, number of failed retries, effective transmission duration, and continuous transmission segment length for the task corresponding to the execution slot within the same sampling window.

[0041] After generating basic operational information, the node further aggregates the operational data of tasks within the same size column to determine the column dissipation difference. Specifically, the node first sums the average connection establishment time of each task within the same size column in the current sampling window by the number of connection establishments, obtaining the total connection establishment time; then it sums the first packet waiting time of each task within the same size column, obtaining the total first packet waiting time; simultaneously, it multiplies the number of failed retries of each task within the same size column by the preset single retry duration, obtaining the total retry duration; subsequently, it sums the effective transmission time of each task within the same size column, obtaining the total effective transmission time. The node then divides the sum of the total connection establishment time, the total first packet waiting time, and the total retry duration by the total effective transmission time to obtain the column dissipation difference of that size column within the current sampling window. Thus, the column dissipation difference can be represented by a specific and calculable ratio, characterizing the occupancy of ineffective transmission behavior relative to effective transmission behavior within the same size column.

[0042] In practice, when there are many tasks in the small file column and connection switching is frequent, the number of connection establishments for tasks within the same size column usually increases, and the total connection establishment time and total first packet waiting time also increase accordingly. If there are also many failure retries, the total retry time will further increase. In this case, even if there is a certain amount of effective transmission time in this size column, its column dissipation difference will be calculated as a high value, indicating that a lot of execution resources in the current size column are consumed in ineffective transmission processes such as connection establishment, waiting for the first packet, and retry recovery. Conversely, when most tasks in a certain size column can quickly complete connection establishment, quickly enter the first packet arrival state, and have few failure retries within the current sampling window, its total effective transmission time accounts for a higher proportion of the total connection establishment time, total first packet waiting time, and total retry time. In this case, the calculated column dissipation difference is lower, indicating that the tasks in this size column are more suitable to continue to maintain or increase execution slots.

[0043] To determine the unit's net transmission continuity length, the node no longer performs overall statistics based on size columns, but instead processes the data on a specific download execution unit basis. Specifically, the node extracts the length of continuous transmission segments for each execution slot within the same download execution unit from entering the effective transmission state to the occurrence of transmission interruption, timeout, or completion within the current sampling window. The node then sums the lengths of each segment to obtain the total continuous transmission length, and simultaneously counts the number of these continuous transmission segments. Subsequently, the node divides the total continuous transmission length by the number of continuous transmission segments to obtain the unit's net transmission continuity length within the current sampling window. This parameter characterizes the average ability of tasks in the current download execution unit to maintain continuous effective transmission once they enter the effective transmission state. A larger unit net transmission continuity length indicates more continuous and stable task transmission within the corresponding download execution unit; a smaller unit net transmission continuity length indicates that tasks in the corresponding download execution unit are more prone to interruption, timeout, or premature termination after entering effective transmission.

[0044] In a specific implementation scenario, if multiple tasks within a small file download execution unit corresponding to a source station row only remain in the effective transmission state for a short period before being interrupted or timed out, the length of the continuous transmission segments formed by each execution slot is generally short, resulting in a small total continuous transmission length, a large number of continuous transmission segments, and a small net transmission continuity length calculated at the end. Conversely, if tasks within a large file download execution unit corresponding to a source station row can continuously occupy the transmission path for a longer period after entering the effective transmission state until completion, the total continuous transmission length is large, the number of continuous transmission segments is relatively small, and the calculated net transmission continuity length at the end of the unit is large. Therefore, the net transmission continuity length at the end of the unit can reflect the continuous transmission capability of the current task set within the download execution unit, providing a direct basis for subsequently determining whether the download execution unit is suitable for continuing to use more execution slots.

[0045] After obtaining the column dissipation difference and the net input continuity length of the cell, the node generates the cell stress relief slot boundary by combining the number of currently enabled slots, failure rate, timeout rate, and the node's remaining bandwidth capacity. Specifically, the node first uses the number of currently enabled slots as the initial slot boundary, then reads the column dissipation difference and the net input continuity length of the cell corresponding to the currently downloading execution unit, and compares them with the preset upper limit of dissipation difference, the preset lower limit of continuity length, the failure rate threshold, the timeout rate threshold, and the preset bandwidth threshold. When the column dissipation difference is less than the preset upper limit of dissipation difference, the unit net output continuous length is greater than the preset lower limit of continuous length, the failure rate and timeout rate are both lower than the corresponding preset thresholds, and the node's remaining bandwidth capacity is higher than the preset bandwidth threshold, it indicates that the current download execution unit is in a relatively optimal operating state. The node increases the initial slot boundary by the preset slot boundary adjustment value to obtain the adjusted unit pressure relief slot boundary. When the column dissipation difference is greater than the preset upper limit of dissipation difference, the unit net output continuous length is less than the preset lower limit of continuous length, the failure rate or timeout rate is higher than the corresponding preset threshold, or the node's remaining bandwidth capacity is lower than the preset bandwidth threshold, it indicates that the current download execution unit is in a state that needs to release pressure. The node subtracts the preset slot boundary adjustment value from the initial slot boundary to obtain the adjusted unit pressure relief slot boundary.

[0046] After the unit pressure relief slot boundary is generated, the node uses it as the direct boundary parameter for subsequent slot control. If the number of currently activated slots for a certain download execution unit is less than the generated unit pressure relief slot boundary, and the column dissipation difference shows a decreasing trend and the unit net transmission continuity length shows an increasing trend in subsequent sampling windows, the node can gradually increase the number of execution slots for that download execution unit without exceeding the unit pressure relief slot boundary. If the number of currently activated slots is greater than the unit pressure relief slot boundary, or the column dissipation difference remains high and the unit net transmission continuity length remains low in subsequent sampling windows, the node stops loading new tasks for that download execution unit and gradually reduces the number of activated slots. In this way, the column dissipation difference, the unit net transmission continuity length, and the unit pressure relief slot boundary form a complete parameter linkage chain: the column dissipation difference reflects the dissipation level in columns of the same size, the unit net transmission continuity length reflects the continuous transmission capability in a specific download execution unit, and the unit pressure relief slot boundary, based on the two and combined with the node's local resource status, forms a unit-level adjustment boundary that can be directly used for execution slot control.

[0047] To ensure the stability of the calculation results in engineering implementation, further settings can be made: when the total effective transmission duration is zero, the column dissipation difference of the corresponding size column is recorded as a preset upper limit value; when the number of consecutive transmission segments is zero, the net continuous transmission length of the corresponding download execution unit is recorded as zero; when the unit pressure relief slot boundary calculated according to the adjustment rules is less than the preset minimum number of slots, the unit pressure relief slot boundary is limited to the preset minimum number of slots. Through the above supplementary processing, the column dissipation difference, the net continuous transmission length of the unit, and the unit pressure relief slot boundary can be prevented from becoming undefined or invalid in extreme cases, further improving the feasibility and operational stability of this technical solution in actual download nodes.

[0048] After calculating the column dissipation difference, the unit net transmission continuity length, and the unit pressure relief slot boundary, the download node enters the dynamic adjustment phase of the execution slots starting from the second sampling window. Specifically, at the end of the current sampling window, the node reads the number of currently active slots of the current download execution unit and compares this value with the unit pressure relief slot boundary generated corresponding to the current sampling window. Simultaneously, the node retrieves the column dissipation difference and the unit net transmission continuity length from the previous and current sampling windows. Subtracting the column dissipation difference of the current sampling window from the column dissipation difference of the previous sampling window yields the dissipation improvement difference, and subtracting the unit net transmission continuity length of the previous sampling window from the unit net transmission continuity length of the current sampling window yields the continuity length increase difference. If the number of currently active slots is less than the unit pressure relief slot boundary, and the dissipation improvement difference is greater than or equal to the preset dissipation difference threshold, and the continuity length increase difference is also greater than or equal to the preset continuity length threshold, it indicates that the current download execution unit has significantly reduced ineffective transmission occupancy and significantly enhanced continuous effective transmission capability without reaching the upper limit of the slot capacity. Based on this, the node executes the operation of increasing the preset number of execution slots.

[0049] During the process of adding execution slots, the node first searches for currently inactive execution slots in the corresponding download execution unit, selects a preset number of execution slots as new execution slots, and switches them to the active state. Then, according to the task priority in the waiting task queue or the order in which tasks entered the waiting task queue, the node selects the material tasks to be downloaded from the waiting task queue of the download execution unit, corresponding to the number of new execution slots, and loads them one by one into the new execution slots for downloading. For each newly loaded task, the node synchronously establishes a new slot operation record, recording the loading start time, connection establishment start time, and current status flag, so that information such as connection establishment, first packet waiting, effective transmission, and continuous transmission segments can be collected in the next sampling window. After the new execution slots are loaded, the node does not immediately expand the slots again. Instead, in the next sampling window, it recalculates the column dissipation difference, the unit net output continuity length, and the unit pressure relief slot boundary before deciding whether to continue adding execution slots, thereby avoiding excessive local fluctuations in the execution unit caused by continuous and rapid slot expansion.

[0050] In another operating scenario, if the number of currently active slots in the current download execution unit is greater than the unit release slot limit, it indicates that the actual number of execution slots activated by the current download execution unit has exceeded the safe upper limit allowed by the current sampling window. Alternatively, if the column dissipation difference of the current sampling window minus the column dissipation difference of the previous sampling window is greater than or equal to the preset dissipation difference threshold, and the unit net input continuity length of the previous sampling window minus the unit net input continuity length of the current sampling window is greater than or equal to the preset continuity length threshold, it indicates that the current download execution unit has experienced significant dissipation degradation and a decrease in continuous transmission capability between the last two sampling windows. In either of these scenarios, the node enters the process of reducing execution slots. At this time, the node no longer loads new downloadable material tasks into the preset number of execution slots in the corresponding download execution unit, but marks these execution slots as pending release and keeps the loaded tasks running until completion. If a loaded task supports breakpoint resumption, the node can also switch its execution state to a suspended state when the preset suspension condition is met, and write the task into the retry task queue, to be reloaded after the subsequent conditions are restored.

[0051] For subsequent processing after reducing execution slots, after a loaded task is completed or suspended, the node sets the corresponding execution slot to an inactive state and synchronously updates the number of currently active slots, the number of waiting tasks, and the number of retry tasks for that download execution unit. If the column dissipation difference continues to decrease in subsequent sampling windows, the unit's net output continuity length increases again, and the unit's pressure relief slot boundary is again higher than the number of currently active slots, the node can restore some inactive execution slots to an active state. Thus, the actions of adding and reducing execution slots are not performed in isolation, but form a closed-loop adjustment mechanism through continuous comparison in adjacent sampling windows, ensuring that the number of execution slots in the same download execution unit always dynamically converges around the unit's pressure relief slot boundary.

[0052] In more complex operational scenarios, nodes continuously assess migration conditions. Specifically, within three consecutive sampling windows, the node compares the column dissipation difference and the net input continuity length of adjacent sampling windows window by window. If the difference between the column dissipation difference of each subsequent sampling window and the difference between the column dissipation difference of the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net input continuity length of the previous sampling window and the net input continuity length of the subsequent sampling window is greater than or equal to a preset continuity length threshold, it indicates that the current download execution unit is in a continuously deteriorating state within three consecutive sampling windows, meaning that ineffective transmission occupancy is constantly increasing and continuous effective transmission capability is constantly weakening. At this point, simply reducing the number of execution slots within the local node is often insufficient to effectively restore the execution state of the download execution unit. Therefore, the node marks the current download execution unit as meeting the migration conditions.

[0053] After the migration conditions are met, the node first selects waiting tasks and retry tasks from the current download execution unit as migration tasks, without directly migrating tasks already in progress. Subsequently, the task scheduling center or the node's local scheduler searches for the target download execution unit in other download nodes. The target download execution unit must meet the following conditions: its source station information is the same as the source station information of the migration task; its file size range is the same as the file size range of the migration task; and the number of currently enabled slots in the target download execution unit is less than its unit pressure relief slot boundary. Once these conditions are met, the migration task is removed from the waiting task queue or retry task queue of the current download execution unit and written to the waiting task queue or retry task queue of the target download execution unit. After writing, the target download node continues to execute the same slot adjustment process in subsequent sampling windows based on its column dissipation difference, unit net output continuity length, and unit pressure relief slot boundary.

[0054] In a specific implementation scenario, if a download execution unit in a download node, targeting a small file column from a certain source station, has a column dissipation difference of a set value and a unit net transmission continuity length of another set value in the first sampling window, and in the second and third sampling windows, the column dissipation difference continuously increases while the unit net transmission continuity length continuously decreases, and the changes between adjacent sampling windows all reach the corresponding preset thresholds, then this download execution unit is determined to meet the migration conditions. At this point, the node selects small file tasks that have not yet started execution and tasks that have failed and been retried but not yet reloaded, and migrates them to download execution units in other download nodes with the same source station, the same file size range, and still having remaining execution capacity. This approach avoids a single download node bearing a set of tasks with high dissipation and low continuous transmission capacity for an extended period, thereby rebalancing the task load across a multi-node cluster.

[0055] Therefore, the three actions—adding execution slots, reducing execution slots, and task migration—are all based on the parameter changes within the sampling window and are uniformly controlled around the unit's pressure relief slot boundaries. Adding execution slots releases the remaining transmission capacity of the current download execution unit, reducing execution slots controls the execution pressure within the current download execution unit, and task migration is used to transfer problematic tasks to other more suitable download nodes when local adjustments fail to recover. The combined effect of these three actions enables the download execution unit to possess both local self-adjustment capabilities and cross-node reallocation capabilities during operation, thereby improving the stability and throughput of the entire multi-node download system in complex mixed-material download scenarios.

[0056] Example 2: This exemplary high-efficiency media download system based on multi-node dynamic concurrency control is used to implement the aforementioned high-efficiency media download method based on multi-node dynamic concurrency control, including: Structure building module: In each download node, a source station row is created according to the source station information and a size column is created according to the file size information. In the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue and multiple execution slots are set up. Task writing module: Writes the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Parameter generation module: Collects the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task in the sampling window. Determines the column dissipation difference based on the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, and effective transmission time for tasks within the same size column. Determines the unit net continuous transmission length based on the continuous transmission segment length of each execution slot in the same download execution unit. Generates the unit pressure relief slot boundary based on the column dissipation difference, unit net continuous transmission length, number of currently enabled slots, failure rate, timeout rate, and remaining bandwidth capacity of the node. Slot control module: Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the column dissipation difference of the previous sampling window minus the column dissipation difference of the current sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the current sampling window minus the unit net output continuity length of the previous sampling window is greater than or equal to a preset continuity length threshold, the number of execution slots is reduced by a preset number when the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the column dissipation difference of the current sampling window minus the column dissipation difference of the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the previous sampling window minus the unit net output continuity length of the current sampling window is greater than or equal to a preset continuity length threshold. When the column dissipation difference of adjacent sampling windows increases window by window and the unit net output continuity length decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

[0057] It should be noted that the efficient material download system based on multi-node dynamic concurrency control provided in the above embodiments and the efficient material download method based on multi-node dynamic concurrency control provided in the above embodiments belong to the same concept. The specific methods of execution of each module and unit have been described in detail in the method embodiments and will not be repeated here. In practical applications, the efficient material download system based on multi-node dynamic concurrency control provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0058] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A high-efficiency material download method based on multi-node dynamic concurrency control, characterized in that, include: Within each download node, a source station row is created based on the source station information, and a size column is created based on the file size information. Within the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue, and multiple execution slots are set up. Write the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Within the sampling window, the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task of the execution slot are collected. The column dissipation difference is determined based on the number of connection establishments, average connection establishment time, first packet wait time, number of failure retries, and effective transmission time for tasks within the same size column. The unit net continuous transmission length is determined based on the continuous transmission segment length of each execution slot within the same download execution unit. The unit pressure relief slot boundary is generated based on the column dissipation difference, the unit net continuous transmission length, the number of currently enabled slots, the failure rate, the timeout rate, and the remaining bandwidth capacity of the node. Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the difference between the column dissipation difference of the previous sampling window and the current sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the current sampling window and the net continuous length of the unit output of the previous sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are added. When the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the difference between the column dissipation difference of the current sampling window and the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the difference between the net continuous length of the unit output of the previous sampling window and the net continuous length of the unit output of the current sampling window is greater than or equal to a preset continuous length threshold, a preset number of execution slots are reduced. When the column dissipation difference of adjacent sampling windows increases window by window and the net continuous length of the unit output decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

2. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, Create source site rows based on source site information and size columns based on file size information, including: Create a corresponding source row based on at least one of the following: source domain name, source address, or source identifier of the material to be downloaded; Establish corresponding size columns based on the preset size range to which the file size of the material to be downloaded belongs; the size columns include at least small file columns, medium file columns, and large file columns. Downloadable materials with the same source site information and file sizes falling within the same preset size range will be grouped into the same download execution unit.

3. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, Write the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information, including: Determine the corresponding source site row based on the source site information of the material task to be downloaded, and determine the corresponding size column based on the file size information of the material task to be downloaded; Write the task of downloading the materials to be downloaded into the waiting task queue of the determined download execution unit; The number of execution slots initially enabled is determined based on the initial capacity of the download node and the number of tasks in the corresponding download execution unit. The material tasks to be downloaded, corresponding to the number of execution slots initially enabled, are retrieved from the waiting task queue and loaded into the execution slots for download.

4. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, Within the sampling window, the following parameters are collected for the task corresponding to the execution slot: number of connection establishments, average connection establishment time, first packet wait time, number of failed retries, effective transmission duration, and length of continuous transmission segments: Record the connection establishment start time, connection establishment completion time, first packet arrival time, entry into valid transmission state time, transmission interruption time, timeout occurrence time, and completion time for the task corresponding to the execution slot; The connection establishment time is determined based on the connection establishment start time and connection establishment completion time; the first packet waiting time is determined based on the connection establishment completion time and the first packet arrival time; and the continuous transmission segment length is determined based on the duration between the time of entering the effective transmission state and the time of transmission interruption, timeout occurrence, or completion. The number of connection establishments and failure retries for each task in each execution slot within the statistical sampling window are recorded.

5. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, The column dissipation difference is determined based on the number of connection establishments, average connection establishment time, first packet wait time, number of retries, and effective transmission time for tasks within the same size column. This includes: The average connection establishment time for tasks within the same size column within the sampling window is summed up by the number of connection establishments to obtain the total connection establishment time; The total first packet waiting time is obtained by summing up the first packet waiting times of tasks within the same size column within the sampling window. Multiply the number of failed retries for tasks within the same size column in the sampling window by the preset single retry duration to obtain the total retry duration; The effective transmission time of tasks within the same size column within the sampling window is summed to obtain the total effective transmission time. The column dissipation difference for this size column is obtained by dividing the sum of the total connection establishment time, the total first packet wait time, and the total retry time by the total effective transmission time.

6. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, The net continuous transmission length of the unit is determined based on the length of the continuous transmission segments of each execution slot within the same download execution unit, including: Extract the length of continuous transmission segments from the entry of a valid transmission state to the occurrence of transmission interruption, timeout, or completion within the sampling window for each execution slot in the same download execution unit; The total continuous transmission length is obtained by summing the lengths of the continuous transmission segments. Count the number of consecutive transmission segments; Divide the total continuous transmission length by the number of continuous transmission segments to obtain the net continuous transmission length of the download execution unit.

7. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, The cell stress relief slot boundaries are generated based on column dissipation difference, net cell input continuity, number of currently active slots, failure rate, timeout rate, and remaining node bandwidth capacity, including: The initial slot boundaries are defined by the number of currently active slots. When the column dissipation difference is less than the preset upper limit of dissipation difference, the net continuous length of the unit is greater than the preset lower limit of continuous length, the failure rate and timeout rate are both lower than the corresponding preset thresholds, and the remaining bandwidth capacity of the node is higher than the preset bandwidth threshold, the initial slot boundary is increased by the preset slot boundary adjustment value to obtain the unit pressure relief slot boundary. When the column dissipation difference is greater than the preset upper limit of dissipation difference, the net continuous length of the unit is less than the preset lower limit of continuous length, the failure rate or timeout rate is higher than the corresponding preset threshold, or the remaining bandwidth capacity of the node is lower than the preset bandwidth threshold, the initial slot boundary is subtracted from the preset slot boundary adjustment value to obtain the unit pressure relief slot boundary.

8. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, The process of adding a preset number of execution slots includes: enabling a preset number of inactive execution slots when the conditions for adding execution slots are met; selecting downloadable material tasks from the waiting task queue of the corresponding download execution unit according to the task priority in the waiting task queue or the order in which they entered the waiting task queue and loading them into the newly added execution slots; and recalculating the column dissipation difference, the unit net output continuity length, and the unit pressure relief slot boundary in the next sampling window after loading the task into the newly added execution slots. Reducing the preset number of execution slots includes: stopping the loading of new downloadable material tasks into the preset number of execution slots in the corresponding download execution unit when the conditions for reducing execution slots are met; keeping the loaded tasks running until completion, or suspending tasks that support resume download and writing them into the retry task queue; and setting the corresponding execution slot to an inactive state after the loaded tasks are completed or suspended.

9. The efficient material download method based on multi-node dynamic concurrency control according to claim 1, characterized in that, Migrate waiting tasks and retry tasks to the corresponding download execution units on other download nodes, including: In three consecutive sampling windows, if the column dissipation difference of each subsequent sampling window minus the column dissipation difference of its preceding sampling window is greater than or equal to a preset dissipation difference threshold, and the net continuous length of the unit input of each preceding sampling window minus the net continuous length of the unit input of each subsequent sampling window is greater than or equal to a preset continuous length threshold, then the current download execution unit is determined to meet the migration conditions. Select waiting tasks and retry tasks as migration tasks from the download execution units that meet the migration conditions; Among other download nodes, select a download execution unit that has the same source station information and file size range as the migration task, and whose currently enabled number of slots is less than its unit release slot boundary, as the target download execution unit, and write the migration task into the target download execution unit's waiting task queue or retry task queue.

10. A high-efficiency material download system based on multi-node dynamic concurrency control, used to implement the high-efficiency material download method based on multi-node dynamic concurrency control as described in any one of claims 1-9, characterized in that, include: Structure building module: In each download node, a source station row is created according to the source station information and a size column is created according to the file size information. In the download execution unit formed by the intersection of the source station row and the size column, a waiting task queue, a retry task queue and multiple execution slots are set up. Task writing module: Writes the materials to be downloaded into the corresponding download execution unit according to the source site information and file size information; Parameter generation module: Collects the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, effective transmission time, and continuous transmission segment length for the corresponding task in the sampling window. Determines the column dissipation difference based on the number of connection establishments, average connection establishment time, first packet waiting time, number of failure retries, and effective transmission time for tasks within the same size column. Determines the unit net continuous transmission length based on the continuous transmission segment length of each execution slot in the same download execution unit. Generates the unit pressure relief slot boundary based on the column dissipation difference, unit net continuous transmission length, number of currently enabled slots, failure rate, timeout rate, and remaining bandwidth capacity of the node. Slot control module: Starting from the second sampling window, when the number of currently enabled slots is less than the unit pressure relief slot boundary, the column dissipation difference of the previous sampling window minus the column dissipation difference of the current sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the current sampling window minus the unit net output continuity length of the previous sampling window is greater than or equal to a preset continuity length threshold, the number of execution slots is reduced by a preset number when the number of currently enabled slots is greater than the unit pressure relief slot boundary, or when the column dissipation difference of the current sampling window minus the column dissipation difference of the previous sampling window is greater than or equal to a preset dissipation difference threshold, and the unit net output continuity length of the previous sampling window minus the unit net output continuity length of the current sampling window is greater than or equal to a preset continuity length threshold. When the column dissipation difference of adjacent sampling windows increases window by window and the unit net output continuity length decreases window by window in three consecutive sampling windows, the waiting task and retry task are migrated to the corresponding download execution unit of other download nodes.

Images