Downloading task execution method and device, equipment, storage medium and program product

Abstract

The invention relates to a downloading task execution method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises the following steps: receiving feedback information sent by a target downloading server, wherein the feedback information comprises resource information and the size of a downloaded file; the target downloading server is any one of at least two downloading servers controlled by a token management server; based on feedback information sent by the target downloading server, token issuing parameters corresponding to the target downloading server are adjusted; and issuing a token to the target downloading server based on the token issuing parameter, so that the target downloading server executes a downloading task based on the received token. Through the method, the downloading tasks of different downloading servers can be integrally controlled, meanwhile, the token issuing parameters are comprehensively adjusted based on the multi-dimensional factors of the downloading servers, the accuracy of the token issuing parameters is improved, and then the control effect of executing the downloading tasks on the downloading servers is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a download task execution method, device, equipment, storage medium and program product. Background Art

[0002] Token bucket is a traffic control algorithm used to limit the access rate to resources. It can be used in network communication, system scheduling, current limiting and other scenarios. At present, improvements to the token bucket algorithm in many scenarios will dynamically adjust the token generation rate and bucket capacity according to traffic conditions to adapt to different traffic patterns and needs. This algorithm can adjust the token generation rate according to real-time traffic conditions, thereby more accurately controlling the rate and delay of traffic.

[0003] However, in related technologies, token generation and consumption services are usually integrated into the same service or module, and the token generation service is deployed on each consumer service node, which makes it impossible to achieve overall control of download tasks of different download servers; and the token generation rate is only adjusted based on traffic conditions, and the regulation method is relatively simple, resulting in poor download task control effect. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device, storage medium and program product for executing a download task, which can realize the overall control of the download tasks of different download servers. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Issuing tokens based on the token issuance parameters can improve the control effect of executing the download task on the download server. The technical solution is as follows.

[0005] On the one hand, a download task execution method is provided, the method is executed by a token management server, the token management server is used to manage and control the download tasks of at least two download servers, the method comprising:

[0006] receiving feedback information sent by a target download server, wherein the feedback information includes resource information and a download file size; the target download server is any one of at least two download servers;

[0007] Based on the feedback information sent by the target download server, adjusting the token issuance parameters corresponding to the target download server;

[0008] A token is issued to the target download server based on the token issuance parameter, so that the target download server executes the download task based on the received token.

[0009] On the other hand, a download task execution method is provided, the method being executed by a target download server, the target download server being any one of at least two download servers for which a token management server performs download task management, the method comprising:

[0010] Receive user download request;

[0011] Sending feedback information of the target download server to the token management server; the feedback information includes resource information and download file size;

[0012] Receive the token issued by the token management server based on the adjusted token issuance parameters, and store the token in a token bucket; the token issuance parameters are obtained by the token management server adjusting the token management server based on the feedback information; the token bucket is a token bucket created and initialized corresponding to the user download request;

[0013] The download task is executed based on the token obtained from the token bucket.

[0014] On the other hand, a download task execution device is provided, the device is applied in a token management server, the token management server is used to manage and control the download tasks of at least two download servers, the device includes:

[0015] An information receiving module, used for receiving feedback information sent by a target download server, wherein the feedback information includes resource information and a download file size; the target download server is any one of at least two download servers;

[0016] A parameter adjustment module, used for adjusting the token sending parameters corresponding to the target download server based on the feedback information sent by the target download server;

[0017] The token issuing module is used to issue a token to the target download server based on the token issuing parameter, so that the target download server executes the download task based on the received token.

[0018] In a possible implementation, the parameter adjustment module is used to:

[0019] Acquiring the feedback information received in the target sliding window;

[0020] Determining average feedback information based on the feedback information received in the target sliding window;

[0021] Based on the average feedback information, the token issuance parameters corresponding to the target download server are adjusted.

[0022] In a possible implementation, the parameter adjustment module includes:

[0023] A first adjustment submodule, used for adjusting the token information corresponding to the token sent to the target download server; the token information includes the download buffer size and the number of download threads;

[0024] and / or,

[0025] The second adjustment submodule adjusts the token generation rate corresponding to the target download server.

[0026] In a possible implementation, the resource information includes CPU utilization information, memory utilization information, and the number of CPU cores; the first adjustment submodule includes:

[0027] A first calculation unit, configured to perform calculation based on the size of the downloaded file and token information of the current token to obtain a linear variable; the linear variable is used to indicate data processing efficiency;

[0028] A second calculation unit is used to calculate based on the number of download threads of the current token, the CPU utilization information and the number of CPU cores of the target download server to obtain a relationship value; the relationship value is used to indicate the correlation between the number of download threads and the CPU utilization; the CPU utilization information includes the current CPU utilization, the maximum CPU utilization and the minimum CPU utilization;

[0029] The first parameter adjustment unit is used to obtain an adjusted download buffer size based on the linear variable, the relationship value and the number of download threads of the current token.

[0030] In a possible implementation, the first adjustment submodule further includes:

[0031] The second parameter adjustment unit is used to adjust the number of download threads based on the current CPU utilization of the target download server; the number of download threads is negatively correlated with the current CPU utilization.

[0032] In a possible implementation, the second adjustment submodule is used to:

[0033] In the case where the size of the downloaded file is smaller than the file size threshold, determining the adjusted token generation rate corresponding to the target download server based on a first calculation method in combination with the resource information and the size of the downloaded file; the first calculation method includes a first smoothing factor constructed based on the remaining token amount and the token consumption rate, and the first smoothing factor is used for global rate adjustment;

[0034] When the downloaded file size is larger than the file size threshold, the adjusted token generation rate corresponding to the target download server is determined based on a second calculation method combined with the resource information and the downloaded file size; the second calculation method includes a second smoothing factor constructed based on the remaining token and the token consumption rate, and the second smoothing factor is used for local rate adjustment.

[0035] In a possible implementation manner, the device further includes:

[0036] An adjustment request receiving module, used for receiving rate adjustment requests fed back by each download server;

[0037] A quantity statistics module, used for counting the number of download servers that initiate the rate adjustment request within a unit time by using a sliding window algorithm;

[0038] The rate adjustment module is used to adjust the token generation rate corresponding to each download server based on the number of the download servers within a unit time.

[0039] In a possible implementation, the rate adjustment module is used to:

[0040] When the rate adjustment request is a request to increase the token generation rate, the proportion of the number of download servers is greater than the first proportion threshold, and the current token generation rate is lower than the maximum token generation rate, the token generation rate corresponding to each download server is increased by a first percentage, and the increased token generation rate is lower than the maximum token generation rate.

[0041] In a possible implementation, the rate adjustment module is used to:

[0042] When the rate adjustment request is a request to reduce the token generation rate, the proportion of the number of download servers is greater than the second proportion threshold, and the current token generation rate is higher than the minimum token generation rate, the token generation rate corresponding to each download server is reduced by a second percentage, and the reduced token generation rate is higher than the minimum token generation rate.

[0043] In a possible implementation, the token issuing module is used to:

[0044] A token generated based on the token sending parameter is sent to the target download server through a message queue.

[0045] On the other hand, a download task execution device is provided, the device is applied in a target download server, the target download server is any one of at least two download servers for download task management and control by a token management server, the device comprising:

[0046] A download request receiving module, used for receiving a user download request;

[0047] An information sending module, used to send feedback information of the target download server to the token management server; the feedback information includes resource information and download file size;

[0048] A token receiving module, used for receiving a token issued by the token management server based on an adjusted token issuance parameter, and storing the token in a token bucket; the token issuance parameter is obtained by the token management server adjusting the token management server based on the feedback information; the token bucket is a token bucket created and initialized corresponding to the user download request;

[0049] The task execution module is used to execute the download task based on the token obtained from the token bucket.

[0050] In a possible implementation, the token receiving module is used to:

[0051] The token corresponding to the target download server and sent by the token management server is received through a message queue.

[0052] On the other hand, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the above-mentioned download task execution method.

[0053] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the above-mentioned download task execution method.

[0054] On the other hand, a computer program product is provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor to implement the download task execution method provided in the above-mentioned various optional implementations.

[0055] The technical solution provided by this application may have the following beneficial effects:

[0056] The download task execution method provided in the embodiment of the present application decouples the token management server from the download server so as to manage the download tasks of multiple download servers through the token management server. Taking the control process of one of the download servers as an example, the token management server adjusts the token issuance parameters corresponding to the target download server based on the feedback information received from the target download server, and issues a token to the target download server based on the token issuance parameters, so that the target download server executes the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above method, the overall control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Sending a token to the download server based on the token issuance parameters can improve the control effect of the download server executing the download task.

[0057] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0059] Figure 1 A flowchart of a download task execution method provided by an exemplary embodiment of the present application is shown;

[0060] Figure 2 A flowchart of a download task execution method provided by an exemplary embodiment of the present application is shown;

[0061] Figure 3 A schematic diagram showing a system structure corresponding to a download task execution method provided by an exemplary embodiment of the present application;

[0062] Figure 4 A flowchart showing a method for executing a download task provided by an exemplary embodiment of the present application;

[0063] Figure 5 A block diagram of a download task execution device provided by an exemplary embodiment of the present application is shown;

[0064] Figure 6 A block diagram of a download task execution device provided by an exemplary embodiment of the present application is shown;

[0065] Figure 7 It is a structural block diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0066] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0067] An embodiment of the present application provides a method for executing a download task, in which a token management server manages and controls the download tasks of a download server by managing and controlling the tokens of the download server, and further manages and controls the download transmission rate of the download server; in an embodiment of the present application, the token management server is separated from the download server to achieve distributed deployment of token generation and token consumption, so that the two parts can be expanded independently. In this case, the token management server can manage and control the download transmission tasks of at least two download servers.

[0068] Figure 1 A flowchart of a download task execution method provided by an exemplary embodiment of the present application is shown. The method can be executed by a token management server, which is used to manage and control the download tasks of at least two download servers. Figure 1 As shown, the method may include the following steps:

[0069] Step 110: receiving feedback information sent by a target download server, the feedback information including resource information and the size of the downloaded file; the target download server is any one of the at least two download servers.

[0070] Each download server obtains its own resource information and download file size in real time and feeds it back to the token management server in the form of feedback information. Correspondingly, the token management server receives the feedback information sent by each download server and issues tokens based on the feedback information of each download server.

[0071] The resource information fed back by the download server may include but is not limited to the resource status of the download server (such as CPU utilization, memory occupancy, etc.), network status, token consumption, etc.; the download file size is used to indicate the file size of the current download task to be downloaded by the download server.

[0072] Step 120: Based on the feedback information sent by the target download server, adjust the token sending parameters corresponding to the target download server.

[0073] After receiving the feedback information sent back by the target download server, the token management server comprehensively adjusts the token issuance parameters corresponding to the target download server based on the above feedback information; since the feedback information sent by different download servers may be different, the token issuance parameters corresponding to different download servers of the token management server are also different.

[0074] In one possible implementation, the target download server can send feedback information to the token management server according to the information feedback cycle. After receiving the feedback information, the token management server adjusts the token issuance parameters based on the received feedback information. In this case, the adjustment frequency of the token issuance parameters is determined by the information feedback cycle of the download server.

[0075] In another possible implementation, the target download server can send feedback information to the token management server according to the information feedback cycle or in real time. After receiving the feedback information, the token management server stores it and adjusts the token issuance parameters corresponding to the target download server according to the parameter adjustment cycle. When the parameter adjustment cycle is reached, the token issuance parameters are comprehensively adjusted based on the feedback information received in the most recent cycle. In this case, the adjustment frequency of the token issuance parameters is determined by the parameter adjustment cycle set in the token management server.

[0076] Step 130: Send a token to a target download server based on the token sending parameter, so that the target download server executes the download task based on the received token.

[0077] Among them, issuing tokens based on token issuing parameters can control the amount of download resources that can be called by the target download server, thereby managing and controlling the download tasks of the target download server, and further, managing and controlling the download service efficiency of the target download server.

[0078] In summary, the download task execution method provided in the embodiment of the present application decouples the token management server from the download server, so as to manage the download tasks of multiple download servers through the token management server. Taking the control process of one of the download servers as an example, the token management server adjusts the token issuance parameters corresponding to the target download server based on the feedback information received from the target download server, and issues a token to the target download server based on the token issuance parameters, so that the target download server executes the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above method, the overall management and control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Sending a token to the download server based on the token issuance parameters can improve the control effect of the download server executing the download task.

[0079] The adjustment of the token issuance parameters may be reflected in the adjustment of the token information corresponding to the token, such as the adjustment of the download buffer size carried by the token and / or the adjustment of the number of download threads; it may also be reflected in the adjustment of the token generation rate. Figure 2 A flowchart of a download task execution method provided by an exemplary embodiment of the present application is shown. The method can be executed by a token management server, which is used to manage and control the download tasks of at least two download servers. Figure 2 As shown, the method includes:

[0080] Step 210: receiving feedback information sent by a target download server, the feedback information including resource information and the size of the downloaded file; the target download server is any one of the at least two download servers.

[0081] In addition, the feedback information may also include token information of the current token, token consumption rate, token remaining amount and other information.

[0082] Taking the example that the adjustment frequency of the token issuance parameters is determined by the parameter adjustment period set in the token management server, in the embodiment of the present application, after receiving the feedback information sent by the target download server, the token management server can extract and integrate the received information based on the sliding window algorithm, so as to adjust the token issuance parameters based on the integrated information. In this case, the token management server can perform the following operations:

[0083] Obtaining feedback information received in the target sliding window;

[0084] Determining average feedback information based on feedback information received within the target sliding window;

[0085] Based on the average feedback information, adjust the token issuance parameters corresponding to the target download server.

[0086] That is to say, the token management server can continuously receive and store the resource information and download file size fed back by the target download server. Each time the token issuance parameters are adjusted, the token management server can extract information based on the target sliding window. Schematically, the target sliding window can be set based on the time length, or the target sliding window can be set based on the number of times the feedback information is received. The size and setting method of the target sliding window can be set based on actual needs, and the present application does not impose any restrictions on this.

[0087] Among them, determining the average feedback information includes averaging the values ​​of resource information received in the target sliding window to determine the average resource information; averaging the sizes of downloaded files received in the target sliding window to determine the average downloaded file size; and adjusting the token download parameters based on the average feedback information.

[0088] In an embodiment of the present application, the token issuance parameters may include at least one of the following: the token generation rate and the token information corresponding to the token; the token information includes the download buffer size and the number of download threads; that is, the token management server can adjust at least one of the above parameters to achieve the adjustment of the token issued to the download server, thereby achieving the control of the download tasks in the download server.

[0089] Step 220: Based on the feedback information sent by the target download server, adjust the token information corresponding to the token sent to the target download server; the token information includes the download buffer size and the number of download threads.

[0090] The resource information in the feedback information includes CPU utilization, memory utilization information, and the number of CPU cores. When the token information is the buffer size, the process of adjusting the token information corresponding to the token sent to the target download server can be implemented as follows:

[0091] Calculate based on the size of the downloaded file and the token information of the current token to obtain a linear variable; the linear variable is used to indicate the data processing efficiency;

[0092] Calculate based on the number of download threads of the current token, CPU utilization information and the number of CPU cores of the target download server to obtain a relationship value; the relationship value is used to indicate the correlation between the number of download threads and the CPU utilization; the CPU utilization information includes the current CPU utilization, the maximum CPU utilization and the minimum CPU utilization;

[0093] Based on the linear variable, the relationship value and the number of download threads of the current token, the adjusted download buffer size is obtained.

[0094] Schematically, the calculation method of the download buffer size f(x)1 can be expressed as:

[0095]

[0096] in, It is a linear variable calculated based on the size of the downloaded file, the buffer size of the current token, and the number of download threads, and is used to indicate the data processing efficiency;

[0097] The relationship curve between the number of threads and CPU utilization is shown. By substituting the number of download threads of the current token, the CPU utilization information and the number of CPU cores of the target download server into the above formula for calculation, the relationship value can be obtained;

[0098] The overall formula for the download buffer size f(x)1 above is expressed as:

[0099]

[0100] Among them, s represents the download file size, x represents the download buffer size, k represents the current CPU utilization, y represents the number of download threads, m represents the maximum CPU utilization, n represents the minimum CPU utilization, and c is the number of CPU cores of the download server; it should be noted that, when the feedback information is integrated based on the target sliding window, the above-mentioned download file size is the average download file size in the target sliding window, and the current CPU utilization information is the average CPU utilization information in the target sliding window.

[0101] When the token information is the number of download threads, the token information corresponding to the token sent to the target download server is adjusted, including:

[0102] The number of download threads is adjusted based on the current CPU utilization of the target download server; the number of download threads is negatively correlated with the current CPU utilization.

[0103] That is, the higher the current CPU utilization, the smaller the number of download threads, wherein the negative correlation between the number of download threads and the current CPU utilization can be set based on actual conditions. In addition, the number of download threads is positively correlated with the number of CPU cores. Since the number of CPU cores of the download server is relatively fixed, for the same download server, when adjusting the number of download threads, the number of CPU cores may not be considered.

[0104] In a possible implementation, when adjusting token information, the number of download threads may be adjusted based on the current CPU utilization, and then the download buffer size may be adjusted based on the adjusted number of download threads, that is, y in the above formula represents the adjusted number of download threads.

[0105] Step 230: Based on the feedback information sent by the target download server, adjust the token generation rate corresponding to the target download server.

[0106] When adjusting the token generation rate, the token management server can determine the token generation rate based on the resource utilization of the download service, the number of CPU cores, the network occupancy rate, the number of download nodes, the size of the download buffer, etc. The function establishes a dynamic balance between multiple factors, while ensuring the download flow limiting effect, and achieving efficient utilization of bandwidth and server resources. The embodiment of the present application takes the adjustment of the token generation rate based on resource information and the size of the downloaded file as an example for explanation.

[0107] In a possible implementation, different token generation rate adjustment methods may be set corresponding to different sizes of downloaded files. Based on this, the process of adjusting the token generation rate corresponding to the target download server may be implemented as follows:

[0108] When the size of the downloaded file is less than the file size threshold, the token generation rate corresponding to the adjusted target download server is determined based on the first calculation method combined with the resource information and the size of the downloaded file; the first calculation method includes a first smoothing factor constructed based on the remaining token amount and the token consumption rate, and the first smoothing factor is used for global rate adjustment;

[0109] When the download file size is larger than the file size threshold, the token generation rate corresponding to the adjusted target download server is determined based on the second calculation method combined with the resource information and the download file size; the second calculation method includes a second smoothing factor constructed based on the remaining token and the token consumption rate, and the second smoothing factor is used for local rate adjustment.

[0110] The file size threshold may be set based on actual needs, and the above-mentioned download file size may refer to the average download file size within the target sliding window.

[0111] Among them, when the size of the downloaded file is less than the file size threshold, it is determined that the current download task corresponds to a small file download scenario. In this scenario, the download server needs to respond quickly and consume tokens frequently to ensure that each small file can be downloaded in time. Therefore, the number of remaining tokens and the consumption rate have a greater impact on the token generation rate. Therefore, the first calculation method includes a first smoothing factor for global rate adjustment; schematically, the calculation formula for the token generation rate in the small file scenario can be expressed as:

[0112] f(x)2=k(A*B)

[0113] in, It is a coefficient integral variable curve of the CPU resource of the download server within the range of maximum and minimum resource utilization, and is used to represent the impact of resource utilization changes on the token generation rate.

[0114] The step value of the token generation rate for the download server's CPU resources in the maximum and minimum resource utilization range.

[0115] k(A*B) is the overall token generation rate curve.

[0116]

[0117] Among them, x is the token generation rate, y is the value of the single-machine resource utilization, n is the minimum value of the single-machine resource utilization, m is the maximum value of the single-machine resource utilization, a is the number of CPU cores of the single machine, t is the download buffer size, j is the JVM memory size, L is the number of download threads, k is the number of download servers, s is the total token consumption rate (percentage), r is the total number of tokens remaining, and p is the overall bandwidth utilization. is the first smoothing factor. By multiplying the entire summation part, the resource utilization at all time points is uniformly adjusted, which can ensure that within the entire time range, the token generation rate can be appropriately adjusted according to the current resource status and request rate, and global rate adjustment can be performed.

[0118] When the size of the downloaded file is larger than the file size threshold, it is determined that the current download task corresponds to a large file download scenario. In this scenario, the download server is more concerned with long-term stability and efficiency. Large file downloads usually take up more resources, such as a larger download buffer and a larger JVM memory. In this case, a stable token generation rate is maintained instead of making large adjustments based on the remaining tokens and the token consumption rate. Therefore, the second calculation method includes a second smoothing factor for local rate adjustment. Schematically, the calculation formula for the token generation rate in the large file scenario can be expressed as:

[0119] f(x)3=k(A*B)

[0120] in, The integral curve of the token generation rate of the machine CPU resources within the maximum and minimum resource utilization range;

[0121] in, is the coefficient step variable of the CPU resource in the maximum and minimum resource utilization range;

[0122] Wherein, k(A*B) is the overall token generation rate curve.

[0123]

[0124] Among them, x is the token generation rate, y is the value of the single-machine resource utilization, n is the minimum value of the single-machine resource utilization, m is the maximum value of the single-machine resource utilization, a is the number of CPU cores of the single machine, t is the download buffer size, j is the JVM memory size, L is the number of threads, k is the number of download servers, s is the total token consumption rate (percentage), r is the total number of tokens remaining, and p is the overall bandwidth utilization. It is the second smoothing factor, which acts in the integral operation and can play a smoothing adjustment role to make the token generation rate relatively stable to meet the demand for long-term stable resource supply required for large file downloads.

[0125] After adjusting the token information and / or the token generation rate of the token in the above manner, an adjusted token issuance parameter is obtained.

[0126] It should be noted that the above steps 220 and 230 can be performed one by one based on actual needs, or they can be applied in combination, and this application does not limit this.

[0127] Step 240: Send a token to the target download server based on the token sending parameter, so that the target download server executes the download task based on the received token.

[0128] The token management server generates a token corresponding to the target download server based on the token delivery parameters, and delivers the token to the target download server.

[0129] In an embodiment of the present application, since the token management server simultaneously manages and controls the download tasks of at least two download servers, the token management server may generate and issue tokens to the corresponding download servers based on the token issuance parameters corresponding to each download server. When the token management server issues tokens, it may issue tokens in the form of a message queue; illustratively, for the target download server, the token management service will issue the token generated based on the token issuance parameters to the target download server through a message queue, so that the target download server executes the download task based on the received token.

[0130] In a possible application scenario, the token management server can also adjust the token generation rate as a whole in a burst scenario, wherein the burst scenario may refer to a sudden decrease or a sudden increase in the task volume. The process can be implemented as follows:

[0131] Receive rate adjustment requests fed back by various download servers;

[0132] The number of download servers that initiate rate adjustment requests per unit time is counted using a sliding window algorithm;

[0133] The token generation rate corresponding to each download server is adjusted based on the number of download servers within a unit time.

[0134] The rate adjustment request fed back by the download server may be a request to increase the token generation rate, or may be a request to reduce the token generation rate.

[0135] Among them, when the sudden scenario is a sudden decrease in the amount of tasks, each download server can quickly complete the download task. At this time, it is necessary to improve resource utilization. When there is no token in the token bucket of the download server or the number of tokens reaches the first number threshold, the download server sends a request to increase the token generation rate to the token management server; the token management server records the number of download servers that initiate the request per unit time through a sliding window algorithm. When the proportion of the number of download servers is greater than the first proportion threshold, and the current token generation rate is lower than the maximum token generation rate, the token generation rate corresponding to each download server is increased by the first percentage, and the increased token generation rate is lower than the maximum token generation rate. Schematically, if more than N% of the download servers initiate a request to increase the token generation rate per unit time, and the current token generation rate corresponding to each download server is lower than the maximum token generation rate, the corresponding token generation rate is increased by S1%, but it is still necessary to ensure that the token generation rate corresponding to each download server after the increase is lower than the maximum token generation rate.

[0136] When the sudden scenario is a sudden increase in the amount of tasks, the resource load of each download server is high, and the number of tokens in the token bucket of most download servers will be higher than the number threshold. In this case, the download server sends a request to the token management server to reduce the token generation rate; the token management server records the number of download servers that initiate the request per unit time through a sliding window algorithm. When the proportion of the number of download servers is greater than the second proportion threshold, and the current token generation rate is higher than the minimum token generation rate, the token generation rate corresponding to each download server is reduced by the second percentage, and the reduced token generation rate is higher than the minimum token generation rate. Schematically, if more than M% of the download servers initiate a request to reduce the token generation rate per unit time, and the token generation rates corresponding to each download server are currently higher than the minimum token generation rate, the corresponding token generation rates are reduced by S2%, but it is still necessary to ensure that the token generation rates corresponding to each download server after the reduction are higher than the minimum token generation rate.

[0137] In summary, the download task execution method provided in the embodiment of the present application decouples the token management server from the download server, so as to manage and control the download tasks of multiple download servers through the token management server. Taking the process of managing and controlling one of the download servers as an example, the token management server adjusts the token issuance parameters corresponding to the target download server based on the feedback information received from the target download server, and issues a token to the target download server based on the token issuance parameters, so that the target download server executes the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above method, the overall management and control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, so as to improve the accuracy of the token issuance parameters. The token is sent to the download server based on the token issuance parameters, which can improve the control effect of the download server executing the download task.

[0138] In addition, in emergency scenarios, the token management server can adjust the overall token generation rate based on the rate adjustment requests received from each download server, thereby achieving overall control over the download servers and improving control efficiency and effectiveness.

[0139] Figure 3 A schematic diagram showing a system structure corresponding to a download task execution method provided by an exemplary embodiment of the present application is shown. Figure 3 As shown, the system includes a token management server 310 and a download server cluster 320, and the download server cluster includes at least two download servers 321; the token management server 310 is used to manage the download tasks of at least two download servers 321, which is manifested as the management and control of the corresponding tokens issued to each download server 321.

[0140] like Figure 3As shown, the token management server may include a token generation component and a rate control component; the rate control component is used to adjust the token issuance parameters based on the feedback information sent by the download server; the token generation component is used to coordinate the task scheduling of each token generation service, manage the initialization configuration information of the token based on the token issuance parameters corresponding to each download management service, and generate and issue the token. Among them, the token management server issues tokens to different download servers through a message queue, and the token corresponding to each download server can be encapsulated into a token message and enter the message queue. The token message may include the buffer size carried by the token and the number of download threads, the number of tokens, and the token array information, etc.; in addition, the token information may also include the identifier of the corresponding download server to complete the corresponding token message distribution; after receiving the token message, the download server parses the token message into a token and stores it in the token bucket corresponding to the download server. The token bucket can adopt the data structure of a blocking queue, and when the token bucket reaches the maximum capacity, the blocking W (ms) is written, and when the token bucket is empty, the blocking R (ms) is read.

[0141] Each download server is also provided with a positive feedback device; the positive feedback device is used to send feedback information to the token management server, and the feedback information may include but is not limited to resource utilization, CPU usage, number of CPU cores, number of download threads, token consumption speed, and bandwidth usage, so that the download service can dynamically adjust the token generation rate through interaction with the token generation service during the user's downloading process. At the same time, in an emergency scenario, the positive feedback device can also feedback the rate adjustment request to the token management server through the token consumption situation.

[0142] Figure 4 A flowchart showing a method for executing a download task provided by an exemplary embodiment of the present application is shown. The method can be executed by a target download server, which is any one of at least two download servers managed by a token management server for download tasks, such as Figure 4 As shown, the method may include the following steps:

[0143] Step 410: receiving a user download request.

[0144] The user download request may include a request instruction, a download file size, download file content, and a user identity.

[0145] Step 420, sending feedback information of the target download server to the token management server; the feedback information includes resource information and the size of the downloaded file.

[0146] After receiving the user's download request, the target download server can create a token bucket and initialize the token bucket; wherein, the token bucket initialization content includes the initialization of token information, that is, the initialization of the token buffer size and the initialization of the number of download threads; the resource utilization, bandwidth utilization, number of CPU cores and other information of the current download server are sent to the token management server. Furthermore, the current token information can also be sent to the token management server, so that the token management server can adjust the token issuance parameters based on the current situation of the download server, and generate and issue tokens based on the adjusted token issuance parameters. The composition of the token management server for adjusting token parameters can refer to the following. Figure 1 or Figure 2 The relevant contents of the illustrated embodiment will not be repeated here.

[0147] Step 430, receiving the token issued by the token management server based on the adjusted token issuance parameters, and storing it in a token bucket; the token issuance parameters are adjusted by the token management server based on feedback information; the token bucket is a token bucket created and initialized in response to a user download request.

[0148] In a possible implementation, the target download server receives the token corresponding to the target download server issued by the token management server through a message queue.

[0149] Among them, the token can be encapsulated in the form of a token message and sent down, and the token message can include the server identifier of the target download server, so that the token message can be sent down to the target download server. The target download server parses the token message to obtain the token and stores it in the token bucket.

[0150] Step 440: Execute the download task based on the token obtained from the token bucket.

[0151] The download service needs to obtain a token from the corresponding token bucket before downloading. The download determines the size of the download buffer and the number of parallel download threads based on the information in the token.

[0152] In summary, the download task execution method provided in the embodiment of the present application decouples the token management server from the download server, so as to manage and control the download tasks of multiple download servers through the token management server. Taking the download task execution process of one of the download servers as an example, after the target download server receives the user's download request, it creates and initializes a token bucket, and sends the feedback information of the target download server to the token management server, so that the token management server adjusts the token issuance parameters based on the feedback information, and issues the token to the target download server based on the token issuance parameters. After receiving the token, the target download server can execute the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above method, the overall management and control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Sending a token to the download server based on the token issuance parameters can improve the control effect of the download server executing the download task.

[0153] Figure 5 A block diagram of a download task execution device provided by an exemplary embodiment of the present application is shown. The device is applied in a token management server and can execute the following steps: Figure 1 or Figure 2 All or part of the steps of the embodiment shown, the token management server is used to manage and control the download tasks of at least two download servers, such as Figure 5 As shown, the device comprises:

[0154] The information receiving module 510 is used to receive feedback information sent by a target download server, wherein the feedback information includes resource information and a download file size; the target download server is any one of at least two download servers;

[0155] A parameter adjustment module 520, configured to adjust a token issuance parameter corresponding to the target download server based on feedback information sent by the target download server;

[0156] The token issuing module 530 is used to issue a token to the target download server based on the token issuing parameter, so that the target download server executes the download task based on the received token.

[0157] In a possible implementation, the parameter adjustment module 520 is used to:

[0158] Acquiring the feedback information received in the target sliding window;

[0159] Determining average feedback information based on the feedback information received in the target sliding window;

[0160] Based on the average feedback information, the token issuance parameters corresponding to the target download server are adjusted.

[0161] In a possible implementation, the parameter adjustment module 520 includes:

[0162] A first adjustment submodule, used for adjusting the token information corresponding to the token sent to the target download server; the token information includes the download buffer size and the number of download threads;

[0163] and / or,

[0164] The second adjustment submodule adjusts the token generation rate corresponding to the target download server.

[0165] In a possible implementation, the resource information includes CPU utilization information, memory utilization information, and the number of CPU cores; the first adjustment submodule includes:

[0166] A first calculation unit, configured to perform calculation based on the size of the downloaded file and token information of the current token to obtain a linear variable; the linear variable is used to indicate data processing efficiency;

[0167] A second calculation unit is used to calculate based on the number of download threads of the current token, the CPU utilization information and the number of CPU cores of the target download server to obtain a relationship value; the relationship value is used to indicate the correlation between the number of download threads and the CPU utilization; the CPU utilization information includes the current CPU utilization, the maximum CPU utilization and the minimum CPU utilization;

[0168] The first parameter adjustment unit is used to obtain an adjusted download buffer size based on the linear variable, the relationship value and the number of download threads of the current token.

[0169] In a possible implementation, the first adjustment submodule further includes:

[0170] The second parameter adjustment unit is used to adjust the number of download threads based on the current CPU utilization of the target download server; the number of download threads is negatively correlated with the current CPU utilization.

[0171] In a possible implementation, the second adjustment submodule is used to:

[0172] In the case where the size of the downloaded file is smaller than the file size threshold, determining the adjusted token generation rate corresponding to the target download server based on a first calculation method in combination with the resource information and the size of the downloaded file; the first calculation method includes a first smoothing factor constructed based on the remaining token amount and the token consumption rate, and the first smoothing factor is used for global rate adjustment;

[0173] When the downloaded file size is larger than the file size threshold, the adjusted token generation rate corresponding to the target download server is determined based on a second calculation method combined with the resource information and the downloaded file size; the second calculation method includes a second smoothing factor constructed based on the remaining token and the token consumption rate, and the second smoothing factor is used for local rate adjustment.

[0174] In a possible implementation manner, the device further includes:

[0175] An adjustment request receiving module, used for receiving rate adjustment requests fed back by each download server;

[0176] A quantity statistics module, used for counting the number of download servers that initiate the rate adjustment request within a unit time by using a sliding window algorithm;

[0177] The rate adjustment module is used to adjust the token generation rate corresponding to each download server based on the number of the download servers within a unit time.

[0178] In a possible implementation, the rate adjustment module is used to:

[0179] When the rate adjustment request is a request to increase the token generation rate, the proportion of the number of download servers is greater than the first proportion threshold, and the current token generation rate is lower than the maximum token generation rate, the token generation rate corresponding to each download server is increased by a first percentage, and the increased token generation rate is lower than the maximum token generation rate.

[0180] In a possible implementation, the rate adjustment module is used to:

[0181] When the rate adjustment request is a request to reduce the token generation rate, the proportion of the number of download servers is greater than the second proportion threshold, and the current token generation rate is higher than the minimum token generation rate, the token generation rate corresponding to each download server is reduced by a second percentage, and the reduced token generation rate is higher than the minimum token generation rate.

[0182] In a possible implementation, the token issuing module 530 is used to:

[0183] A token generated based on the token sending parameter is sent to the target download server through a message queue.

[0184] In summary, the download task execution device provided in the embodiment of the present application decouples the token management server from the download server, so as to manage and control the download tasks of multiple download servers through the token management server. Taking the process of managing and controlling one of the download servers as an example, the token management server adjusts the token issuance parameters corresponding to the target download server based on the feedback information received from the target download server, and issues a token to the target download server based on the token issuance parameters, so that the target download server executes the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above-mentioned device, the overall management and control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Sending a token to the download server based on the token issuance parameters can improve the control effect of the download server executing the download task.

[0185] Figure 6 A block diagram of a download task execution device provided by an exemplary embodiment of the present application is shown. The device is applied in a target download server and can execute the following steps: Figure 4 In all or part of the steps of the embodiment shown, the target download server is any one of the at least two download servers for which the token management server performs download task management, such as Figure 6 As shown, the device comprises:

[0186] The download request receiving module 610 is used to receive a user download request;

[0187] An information sending module 620 is used to send feedback information of the target download server to the token management server; the feedback information includes resource information and download file size;

[0188] The token receiving module 630 is used to receive the token issued by the token management server based on the adjusted token issuance parameters and store it in a token bucket; the token issuance parameters are adjusted by the token management server based on the feedback information; the token bucket is a token bucket created and initialized corresponding to the user download request;

[0189] The task execution module 640 is used to execute the download task based on the token obtained from the token bucket.

[0190] In a possible implementation, the token receiving module 630 is used to:

[0191] The token corresponding to the target download server and sent by the token management server is received through a message queue.

[0192] In summary, the download task execution device provided in the embodiment of the present application decouples the token management server from the download server so as to manage and control the download tasks of multiple download servers through the token management server. Taking the download task execution process of one of the download servers as an example, based on the device, after the target download server receives the user's download request, it creates and initializes a token bucket, and sends the feedback information of the target download server to the token management server, so that the token management server adjusts the token issuance parameters based on the feedback information, and issues the token to the target download server based on the token issuance parameters. After receiving the token, the target download server can execute the download task based on the received token; wherein the feedback information includes resource information and download file information. Through the above-mentioned device, the overall management and control of the download tasks of different download servers can be achieved. At the same time, the token issuance parameters are comprehensively adjusted based on the multi-dimensional factors of the download server, thereby improving the accuracy of the token issuance parameters. Sending tokens to the download server based on the token issuance parameters can improve the control effect of the download server executing the download task.

[0193] Figure 7 The structural block diagram of a computer device 700 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as a token management server and a download server in the above scheme of the present application. The computer device 700 includes a central processing unit (CPU) 701, a system memory 704 including a random access memory (RAM) 702 and a read-only memory (ROM) 703, and a system bus 705 connecting the system memory 704 and the central processing unit 701. The computer device 700 also includes a large-capacity storage device 706 for storing an operating system 709, an application program 710 and other program modules 711.

[0194] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 704 and mass storage device 706 can be collectively referred to as memory.

[0195] According to various embodiments of the present disclosure, the computer device 700 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 700 can be connected to the network 708 through the network interface unit 707 connected to the system bus 705, or the network interface unit 707 can be used to connect to other types of networks or remote computer systems (not shown).

[0196] The memory also includes at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is stored in the memory. The central processing unit 701 implements all or part of the steps in the download task execution method shown in the above-mentioned embodiments by executing the at least one instruction, at least one program, code set or instruction set.

[0197] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above-mentioned download task execution method. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0198] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer, the computer executes the above Figure 1 , Figure 2 or Figure 4 All or part of the steps of the download task execution method shown in any embodiment.

[0199] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0200] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A download task execution method, characterized in that: The method is performed by a token management server, and the token management server is used to manage and control the download tasks of at least two download servers. The method includes: receiving feedback information sent by a target download server, wherein the feedback information includes resource information and a download file size; the target download server is any one of at least two download servers; Based on the feedback information sent by the target download server, adjusting the token issuance parameters corresponding to the target download server; A token is issued to the target download server based on the token issuance parameter, so that the target download server executes the download task based on the received token.

2. The method according to claim 1, characterized in that The adjusting the token issuing parameters corresponding to the target download server based on the resource information and the download file size fed back by the target download server includes: Acquiring the feedback information received in the target sliding window; Determining average feedback information based on the feedback information received in the target sliding window; Based on the average feedback information, the token issuance parameters corresponding to the target download server are adjusted.

3. The method according to claim 1 or 2, characterized in that: The step of adjusting the token issuing parameters corresponding to the target download server includes: Adjusting the token information corresponding to the token sent to the target download server; the token information includes the download buffer size and the number of download threads; and / or, Adjust the token generation rate corresponding to the target download server.

4. The method according to claim 3, characterized in that: The resource information includes CPU utilization information, memory utilization information and the number of CPU cores; The token information corresponding to the token sent to the target download server is adjusted, including: Calculate based on the size of the downloaded file and the token information of the current token to obtain a linear variable; the linear variable is used to indicate data processing efficiency; Calculate based on the number of download threads of the current token, the CPU utilization information and the number of CPU cores of the target download server to obtain a relationship value; the relationship value is used to indicate the correlation between the number of download threads and the CPU utilization; the CPU utilization information includes the current CPU utilization, the maximum CPU utilization and the minimum CPU utilization; Based on the linear variable, the relationship value and the number of download threads of the current token, an adjusted download buffer size is obtained.

5. The method according to claim 3, characterized in that: The adjusting the token information corresponding to the token issued to the target download server also includes: The number of download threads is adjusted based on the current CPU utilization of the target download server; the number of download threads is negatively correlated with the current CPU utilization.

6. The method according to claim 3, characterized in that The adjusting the token generation rate corresponding to the target download server includes: In the case where the size of the downloaded file is smaller than the file size threshold, determining the adjusted token generation rate corresponding to the target download server based on a first calculation method in combination with the resource information and the size of the downloaded file; the first calculation method includes a first smoothing factor constructed based on the remaining token amount and the token consumption rate, and the first smoothing factor is used for global rate adjustment; When the downloaded file size is larger than the file size threshold, the adjusted token generation rate corresponding to the target download server is determined based on a second calculation method combined with the resource information and the downloaded file size; the second calculation method includes a second smoothing factor constructed based on the remaining token and the token consumption rate, and the second smoothing factor is used for local rate adjustment.

7. The method according to claim 1, characterized in that The method further comprises: Receive rate adjustment requests fed back by various download servers; Counting the number of download servers that initiate the rate adjustment request within a unit time by using a sliding window algorithm; The token generation rate corresponding to each download server is adjusted based on the number of download servers within a unit time.

8. The method according to claim 7, characterized in that In the case where the rate adjustment request is a request to increase the token generation rate, adjusting the token generation rate corresponding to each download server based on the number of download servers within a unit time includes: When the proportion of the number of download servers is greater than the first proportion threshold and the current token generation rate is lower than the maximum token generation rate, the token generation rate corresponding to each download server is increased by a first percentage, and the increased token generation rate is lower than the maximum token generation rate.

9. The method according to claim 7, characterized in that: In the case where the rate adjustment request is a request to reduce the token generation rate, adjusting the token generation rate corresponding to each download server based on the number of download servers within a unit time includes: When the proportion of the number of download servers is greater than the second proportion threshold and the current token generation rate is higher than the minimum token generation rate, the token generation rate corresponding to each download server is reduced by a second percentage, and the reduced token generation rate is higher than the minimum token generation rate.

10. The method according to claim 1, characterized in that The sending of a token to the target download server based on the token sending parameter includes: A token generated based on the token sending parameter is sent to the target download server through a message queue.

11. A download task execution method, characterized in that: The method is executed by a target download server, which is any one of at least two download servers for which a token management server performs download task management and control, and includes: Receive download request from user; Sending feedback information of the target download server to the token management server; the feedback information includes resource information and download file size; Receive the token issued by the token management server based on the adjusted token issuance parameters, and store the token in a token bucket; the token issuance parameters are obtained by the token management server adjusting the token management server based on the feedback information; the token bucket is a token bucket created and initialized corresponding to the user download request; The download task is executed based on the token obtained from the token bucket.

12. The method according to claim 11, characterized in that The receiving a token issued by the token management server based on the adjusted token issuance parameter includes: The token corresponding to the target download server and sent by the token management server is received through a message queue.

13. A download task execution device, characterized in that: The device is applied in a token management server, and the token management server is used to manage and control the download tasks of at least two download servers. The device includes: An information receiving module, used for receiving feedback information sent by a target download server, wherein the feedback information includes resource information and a download file size; the target download server is any one of at least two download servers; A parameter adjustment module, used for adjusting the token sending parameters corresponding to the target download server based on the feedback information sent by the target download server; The token issuing module is used to issue a token to the target download server based on the token issuing parameter, so that the target download server executes the download task based on the received token.

14. A download task execution device, characterized in that: The device is applied in a target download server, which is any one of at least two download servers for which a token management server performs download task management and control, and includes: A download request receiving module, used for receiving a user download request; An information sending module, used to send feedback information of the target download server to the token management server; the feedback information includes resource information and download file size; A token receiving module, used for receiving a token issued by the token management server based on an adjusted token issuance parameter, and storing the token in a token bucket; the token issuance parameter is obtained by the token management server adjusting the token management server based on the feedback information; the token bucket is a token bucket created and initialized corresponding to the user download request; The task execution module is used to execute the download task based on the token obtained from the token bucket.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the download task execution method as described in any one of claims 1 to 12.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by the processor to implement the download task execution method as described in any one of claims 1 to 12.

17. A computer program product, characterized in that The computer program product comprises at least one computer program, and the computer program is loaded and executed by a processor to implement the download task execution method according to any one of claims 1 to 12.

Images