Resource adjustment method, device, equipment and storage medium for Flink task processing

Abstract

The present application relates to a resource adjustment method, device, equipment and storage medium for Flink task processing. The method includes: obtaining the actual usage of each resource usage indicator during the execution of the current Flink task; generating a total resource usage score of the current Flink task according to each actual usage and the indicator parameters of each resource usage indicator; obtaining the adjustment strategy of the current Flink task according to the total resource usage score and the mapping relationship between the total resource usage score and the adjustment strategy; adjusting the resource allocation of the current Flink task according to the adjustment strategy. The present application scores the resource usage of the current Flink task from multiple dimensions by setting multiple resource usage indicators, and accurately locates the adjustment strategy that matches the current Flink task based on the mapping relationship between the total resource usage score and the adjustment strategy, so that the acquisition of the adjustment strategy is more accurate, and the resources are adjusted based on the adjustment strategy, so that the adjustment of resources is more accurate.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a resource adjustment method, device, equipment and storage medium for Flink task processing. Background Art

[0002] At present, each TaskManager (task manager) of the underlying Flink framework is set to 1C and 2G by default. Users can adjust the number of slots (task slots) and memory of TaskManager through advanced parameters. For example, users currently set concurrency mainly by setting multiple TaskManagers or setting multiple slots through advanced parameters. The main problem is that when users configure concurrency, they lack experience in resource configuration. It is often the case that problems can be solved with very little concurrency. As a result, the concurrency is too large, the CPU and memory utilization is low, and resources are wasted. Therefore, how to reasonably adjust resources has become a problem to be solved.

[0003] In traditional technical solutions, when the Flink framework processes tasks, resource utilization is generally evaluated through the maximum and average QPS (throughput) values, and resources are adjusted based on the evaluation results. This can easily lead to inaccurate evaluation and unreasonable adjustments. Summary of the invention

[0004] Based on this, it is necessary to provide a resource adjustment method, device, equipment and storage medium for Flink task processing in response to the above technical problems.

[0005] A resource adjustment method for Flink task processing, the method comprising:

[0006] Get the actual usage of each preset resource usage indicator during the execution of the current Flink task;

[0007] Generate the total resource usage score of the current Flink task based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator;

[0008] According to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy, the adjustment strategy of the current Flink task is obtained;

[0009] Adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0010] In one embodiment, the indicator parameters include a maximum value and a maximum indicator score. The total resource usage score of the current Flink task is generated based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator, including:

[0011] When the evaluation factor to which the resource usage indicator belongs is the first factor, the usage ratio of the resource usage indicator of each first factor is generated according to the actual usage and the maximum value of the resource usage indicator of each first factor;

[0012] Generate a first score based on the usage ratio of the resource usage indicators of each first factor and the highest score of the indicator;

[0013] When the evaluation factor to which the resource usage indicator belongs is the second factor, the resource usage indicator includes the first indicator and the second indicator, and the consumption proportion is generated according to the actual usage of the first indicator and the actual usage of the second indicator;

[0014] Generate a second score based on the consumption proportion and the highest score of the resource use indicator of the second factor;

[0015] A total resource usage score of the current Flink task is generated based on the second score and each first score.

[0016] In one embodiment, the above-mentioned mapping relationship between the total resource usage score of the current Flink task and the pre-configured total resource usage score and the adjustment strategy is used to obtain the adjustment strategy of the current Flink task, including:

[0017] According to the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy matching the interval to which the total resource usage score of the current Flink task belongs is obtained, and the adjustment strategy of the current Flink task is obtained.

[0018] In one of the embodiments, the adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted, and adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task includes:

[0019] Adjust the resource allocation corresponding to each resource usage indicator of the current Flink task according to the adjustment value corresponding to the resource usage indicator to be adjusted.

[0020] In one embodiment, the method further comprises:

[0021] When receiving the resource adjustment request sent by the terminal, the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is executed;

[0022] Alternatively, when it is monitored that the current Flink task starts to run, a step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is performed.

[0023] In one embodiment, the method further comprises:

[0024] Compare the actual usage corresponding to each resource usage indicator with the preset threshold to obtain a comparison result;

[0025] Generate resource usage details information corresponding to each resource usage indicator according to the comparison results;

[0026] Send resource usage details to the terminal for display.

[0027] In one embodiment, the method further comprises:

[0028] Receive a configuration request for a mapping relationship between a total resource usage score and an adjustment strategy;

[0029] According to the newly added adjustment policy in the configuration request and the total resource usage score corresponding to the newly added adjustment policy, a mapping relationship between the newly added adjustment policy and the total resource usage score corresponding to the newly added adjustment policy is created.

[0030] A resource adjustment device for Flink task processing, the device comprising:

[0031] The acquisition module is used to obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task;

[0032] The generation module is used to generate the total resource usage score of the current Flink task based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator;

[0033] A processing module is used to obtain the adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy;

[0034] The adjustment module is used to adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0035] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0036] Get the actual usage of each preset resource usage indicator during the execution of the current Flink task;

[0037] Generate the total resource usage score of the current Flink task based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator;

[0038] According to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy, the adjustment strategy of the current Flink task is obtained;

[0039] Adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task

[0040] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0041] Get the actual usage of each preset resource usage indicator during the execution of the current Flink task;

[0042] Generate the total resource usage score of the current Flink task based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator;

[0043] According to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy, the adjustment strategy of the current Flink task is obtained;

[0044] Adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0045] The resource adjustment method, device, computer equipment and storage medium for the above-mentioned Flink task processing obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task; generate the total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the indicator parameters of each pre-configured resource usage indicator; obtain the adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy; adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task. This application scores the resource usage of the current Flink task from multiple dimensions by setting multiple resource usage indicators, and based on the mapping relationship between the pre-configured total resource usage score and the adjustment strategy, accurately locates the adjustment strategy that matches the current Flink task according to the obtained score, so that the acquisition of the adjustment strategy is more accurate, and the resources are adjusted based on the adjustment strategy, so that the adjustment of resources is more reasonable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is an application environment diagram of a resource adjustment method for Flink task processing in one embodiment;

[0047] Figure 2 A flowchart of a resource adjustment method for Flink task processing in another embodiment;

[0048] Figure 3 A schematic diagram of a detailed flow chart of steps for generating a total resource usage score of a current Flink task according to actual usage of each resource usage indicator and pre-configured indicator parameters of each resource usage indicator in an embodiment;

[0049] Figure 4 A flowchart of a resource adjustment method for Flink task processing in another embodiment;

[0050] Figure 5 A structural block diagram of a resource adjustment device for Flink task processing in one embodiment;

[0051] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Please refer to Figure 1 , Figure 1 A schematic diagram of an application environment of a resource adjustment method for Flink task processing provided by an exemplary embodiment of the present application. Figure 1 As shown, the application environment includes a server 100 and a terminal 101. The server 100 and the terminal 101 can be communicatively connected via a network 102 to implement the resource adjustment method for Flink task processing of the present application.

[0054] The server 100 is used to obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task when receiving the resource adjustment request sent by the terminal 101; generate the total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator; obtain the adjustment strategy of the current Flink task according to the mapping relationship between the total resource usage score of the current Flink task and the pre-configured total resource usage score and the adjustment strategy; adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task, and generate resource usage details information corresponding to each resource usage indicator according to the actual usage and indicator score corresponding to each resource usage indicator; send the resource usage details information to the terminal 101 for display. Among them, the server 100 can be implemented by an independent server or a server cluster composed of multiple servers.

[0055] The terminal 101 is used to send a resource adjustment request to the server 100, and receive resource usage details corresponding to each resource usage indicator sent by the server, and display the resource usage details corresponding to each resource usage indicator on the user interface. The terminal 101 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0056] The network 102 is used to realize the network connection between the terminal 101 and the server 100. Specifically, the network 102 may include various types of wired or wireless networks.

[0057] In one embodiment, Figure 2 As shown in the figure, a resource adjustment method for Flink task processing is provided. Figure 1 The server in the example is used to illustrate the following steps:

[0058] S11. Obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task.

[0059] In this application, the above-mentioned current Flink task can be a real-time data stream processing task of the Flink framework. The above-mentioned resource usage indicators may include the highest memory usage, average memory usage, FullGC (full garbage collection) times, FullGC time, the highest TaskManager (task manager) CPU usage, the average TaskManager CPU usage, the task (task) sender Buffer (buffer) usage, the task sender Buffer usage, committedOffsets (committed offset (compensation) to a specific location), and currentOffsets (current offset position).

[0060] Furthermore, the actual usage of the above-mentioned resource usage indicators is the actual value of each resource usage indicator during the operation of the current Flink task. For example, the maximum memory usage can be obtained during the operation of the current Flink task. Specifically, the present application can pre-set a preset time period and obtain the actual usage of each resource usage indicator within the preset period. For example, if the preset time period is one hour, the actual usage of each resource usage indicator within one hour is obtained. It can also be set to obtain the actual usage of each resource usage indicator within a certain time period of a certain day. The specific situation is set according to actual needs and is not specifically limited here.

[0061] S12. Generate a total resource usage score for the current Flink task based on the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator.

[0062] In this application, different indicator parameters are set for different resource usage indicators. For example, the indicator parameters for average memory usage can be set as a maximum memory value of 16 and a maximum indicator score of 50.

[0063] In this application, the score of each resource usage indicator can be obtained according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator, and the total resource usage score of the current Flink task can be obtained further according to the score of each resource usage indicator.

[0064] In one embodiment, the above-mentioned indicator parameters include a maximum value and a highest indicator score, such as Figure 3 As shown, a detailed flowchart of the steps for generating the total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator is provided, including the following steps:

[0065] S31. When the evaluation factor to which the resource usage indicator belongs is the first factor, generating a usage ratio of the resource usage indicator of each first factor according to the actual usage amount and the maximum value of the resource usage indicator of each first factor;

[0066] S32, generating a first score according to the usage proportion of the resource usage indicator of each first factor and the highest score of the indicator;

[0067] S33: When the evaluation factor to which the resource usage indicator belongs is the second factor, the resource usage indicator includes the first indicator and the second indicator, and a consumption proportion is generated according to an actual usage of the first indicator and an actual usage of the second indicator;

[0068] S34, generating a second score according to the consumption proportion and the highest score of the resource use indicator of the second factor;

[0069] S35. Generate a total resource usage score of the current Flink task based on the second score and each first score.

[0070] In the present application, multiple evaluation factors are pre-set. The evaluation factors may include memory, CPU, FullGC, Kafka consumption capacity, back pressure, data skew, etc. One or more resource usage indicators may be set for each evaluation factor. For example, corresponding resource usage indicators may be set for memory, including the maximum memory usage and the average memory usage. Corresponding resource usage indicators may be set for CPU, including the maximum TaskManagerCPU usage and the average TaskManagerCPU usage. Corresponding resource usage indicators may be set for FullGC, including the number of FullGC times and the FullGC time. Corresponding resource usage indicators may be set for Kafka consumption capacity, including committedOffsets and currentOffsets. Corresponding resource usage indicators may be set for back pressure and data skew, including the usage of the task sending end Buffer and the usage of the task receiving end Buffer.

[0071] Furthermore, the first factor includes memory, CPU, FullGC, back pressure, and data skew. The second factor includes Kafka consumption capacity. The first indicator is committedOffsets. The second indicator is currentOffsets.

[0072] Specifically, for the memory evaluation factor, assuming that the indicator parameters corresponding to the memory are set to include a maximum memory value of 16, a maximum indicator score of 50, and the highest memory usage under the memory evaluation factor is M, then the score of the memory evaluation factor = M / 16*50.

[0073] Similarly, for the CPU evaluation factor, assuming that the indicator parameters corresponding to the CPU are set to include the maximum CPU value of 100, the highest indicator score of 20, and the average TaskManager CPU usage under the CPU evaluation factor is obtained as N. Then the score of the CPU evaluation factor = N / 100*20.

[0074] For the FullGC evaluation factor, assume that the indicator parameters corresponding to FullGC include the maximum value of the FullGC times as 20, the highest indicator score is 5, and the actual value of the FullGC times is S. Then the score of the FullGC evaluation factor = S / 20*5.

[0075] For the back pressure and data skew assessment factors, assume that the corresponding indicator parameters are set to include a maximum score of 5 for back pressure and a maximum score of 100. The actual value of the buffer usage at the task sender is obtained as Y. Then the score of the back pressure and data skew assessment factor = Y / 100*5.

[0076] For the Kafka consumption capacity evaluation factor, assume that the corresponding indicator parameter is set to offset with a maximum score of 15. The actual value of committedOffsets is E, and the actual value of currentOffsets is F. The score of the Kafka consumption capacity evaluation factor = F / E*15.

[0077] Therefore, the first score mentioned above includes the score of the memory evaluation factor = M / 16*50, the score of the CPU evaluation factor = N / 100*20, the score of the FullGC evaluation factor = S / 20*5, and the score of the back pressure and data skew evaluation factor = Y / 100*5. The second score mentioned above is the score of the Kafka consumption capacity evaluation factor = F / E*15. The calculation formula for the total resource usage score of the current Flink task is as follows:

[0078] Total resource usage score = M / 16*50+N / 100*20+S / 20*5+Y / 100*5+F / E*15

[0079] In this application, by setting multiple resource usage indicators and obtaining the actual usage of each resource usage indicator in any time period, the total resource usage score of the current Flink task is generated based on the actual usage, thereby realizing the evaluation of resource utilization from multiple dimensions, making the evaluation of resource utilization more accurate and scientific, and further making the adjustment of resource utilization more reasonable, solving the problem of resource waste caused by unreasonable resource allocation in the prior art, and improving resource utilization.

[0080] It should be noted that there may be multiple resource usage indicators for one evaluation factor. In this case, when calculating the total resource usage score of the current Flink task, the actual usage of one resource usage indicator can be selected from the multiple resource usage indicators for calculation. For example, there are two resource usage indicators, namely, the highest memory usage and the average memory usage, under the memory evaluation factor. When calculating the total resource usage score of the current Flink task, the actual value of the highest memory usage can be selected for calculation.

[0081] In another embodiment, all resource usage indicators under each evaluation factor may also be used in the calculation. For example, the first scores corresponding to the highest memory usage and the average memory usage may be used in the calculation of the total resource usage score of the current Flink task.

[0082] S13. Obtain the adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy.

[0083] In the present application, the mapping relationship between the above-mentioned total resource usage score and the adjustment strategy includes the corresponding relationship between the total resource usage score and the adjustment strategy. Among them, the above-mentioned adjustment strategy may include the resource usage indicator to be adjusted and the corresponding adjustment value, and may also include the reasons for insufficient resources or excessive resources, and may also include the usage details of other resource usage indicators.

[0084] Specifically, the present application can obtain an adjustment strategy based on the total resource usage score of the current Flink task, and can also obtain a corresponding adjustment strategy based on the scores of each evaluation factor of the current Flink task, so that the obtained adjustment strategy is more accurate.

[0085] For example, the total resource usage score of the current Flink task is A. According to the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy is: it is recommended to reduce the CPU to 2C and the memory to 4G.

[0086] In one embodiment, the above-mentioned obtaining the adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy may include:

[0087] According to the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy matching the interval to which the total resource usage score of the current Flink task belongs is obtained, and the adjustment strategy of the current Flink task is obtained.

[0088] In the present application, the mapping relationship between the above-mentioned total resource usage score and the adjustment strategy includes the corresponding relationship between the total resource usage score and the adjustment strategy. Among them, it can be a one-to-one correspondence between the total resource usage score and the adjustment strategy, or it can be a one-to-one correspondence between the numerical range to which the total resource usage score belongs and the adjustment strategy. The former is that one score corresponds to one adjustment strategy, and the latter is that one score range corresponds to one adjustment strategy. The specific settings can be set according to actual needs.

[0089] Through this implementation, the present application can locate the adjustment strategy more accurately, making the adjustment of resources more precise.

[0090] S14. Adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0091] In the present application, the above adjustment strategy may include the resource usage indicator to be adjusted and the corresponding adjustment value, and may also include the reasons leading to insufficient resources or excessive resources, and may also include detailed usage information of other resource usage indicators.

[0092] In one embodiment, the adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted. Adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task may include:

[0093] Adjust the resource allocation corresponding to each resource usage indicator of the current Flink task according to the adjustment value corresponding to the resource usage indicator to be adjusted.

[0094] In this application, the adjustment strategy of the current Flink task includes the adjustment value corresponding to the resource usage indicator to be adjusted and the resource usage details information of each resource usage indicator.

[0095] For example, the above adjustment strategy can be:

[0096] The resources currently applied for by Task B are 4C of CPU and 8G of memory. During the statistical period (2021-11-8:10 to 2021-11-8:11), the total score is C. According to the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy is matched as follows: the actual resource usage rate is low, and it is recommended to reduce the resources to 2C (CPU) 4G (memory).

[0097] Among them, the resource usage details are as follows:

[0098] (1) Memory: The memory requested is 8 GB, and the actual average memory usage during the statistical period is 3 GB.

[0099] (2)CPU: 4C was applied for, and 2 was actually used during the statistical period.

[0100] (3) FullGC: During the statistical period, FullGC occurred twice, with a total duration of 5 minutes, which is not frequent.

[0101] (4) Back pressure and data skew: The average usage of outPoolUsage and inPoolUsage during the statistical period is less than 30%, and no back pressure occurs.

[0102] (5) Consumption capacity: Offset consumption capacity is normal and there is no backlog.

[0103] According to the above adjustment strategy, it can be known that the adjustment strategy includes the adjustment value of the resource usage indicator to be adjusted and the resource usage details information. Among them, the resource usage details information describes the actual usage information of each resource usage indicator and whether it is within a reasonable range. According to the above adjustment strategy, the memory can be reduced to 4G and the CPU can be reduced to 2C. The present application can realize customized adjustment of resources according to the adjustment strategy, which is highly flexible.

[0104] In one embodiment, the above method may further include:

[0105] Upon receiving the resource usage adjustment request sent by the terminal, the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is executed;

[0106] Alternatively, when it is monitored that the current Flink task starts to run, a step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is performed.

[0107] This application can trigger the resource adjustment method for Flink task processing described above through an adjustment request submitted by the user end after the current Flink task is created. The resource adjustment method for Flink task processing can also be automatically triggered when the current Flink task starts running. Generally, each Flink task can execute the resource adjustment method for Flink task processing described above once during execution. Specifically, when obtaining the actual usage of each resource usage indicator, the actual usage in any time period during the task execution process can be obtained, or the actual usage of the entire process can be obtained.

[0108] This application obtains the actual usage of each resource usage indicator within any time period, and generates an adjustment strategy based on the actual usage of each resource usage indicator within the arbitrary time period, so that the adjustment strategy is more accurate and the flexibility of the solution is also improved.

[0109] In one embodiment, if Figure 4As shown, the above method may further include:

[0110] S41, comparing the actual usage corresponding to each resource usage indicator with a preset threshold value to generate a comparison result;

[0111] S42, generating resource usage details information corresponding to each resource usage indicator according to the comparison result;

[0112] S43: Send resource usage details to the terminal for display.

[0113] The resource usage details include the actual usage of each resource usage indicator, whether each actual usage is within a reasonable range, and the corresponding cause analysis.

[0114] The resource usage details information corresponding to each resource usage indicator is generated according to the actual usage amount and indicator score corresponding to each resource usage indicator, which may include:

[0115] The actual usage corresponding to each resource usage indicator is compared with the preset threshold, and resource usage details are generated based on the comparison results.

[0116] In this application, the actual usage corresponding to each resource usage indicator can be compared with the preset threshold, and resource usage details information can be generated according to the comparison result. In another embodiment, the actual usage corresponding to each resource usage indicator can also be converted into a usage rate, and the obtained usage rate can be compared with the preset threshold.

[0117] Specifically, when the resource usage indicator is the highest memory usage, the highest memory usage rate is obtained according to the highest memory usage, and the highest memory usage rate is compared with the corresponding preset threshold. It is determined whether the above-mentioned highest memory usage rate is lower than the preset threshold. If so, the memory usage rate of the entire task is low, and it can be recommended to reduce the memory of TaskManager or reduce the number of TaskManagers. The above-mentioned preset threshold can be 30%.

[0118] When the resource usage indicator is the number of FullGCs, the corresponding preset threshold can be 50%. Specifically, the number of FullGCs and the duration of each FullGC in a preset time period, such as an hour, can be counted, and the product of the number of FullGCs and the duration of each FullGC can be compared with the statistical period. If it exceeds 50% of the statistical period, it is generated that the FullGC is too frequent during the execution of the current Flink task, and the resource usage details of the TaskManager memory can be considered to be improved.

[0119] When the resource usage indicator is the highest TaskManagerCPU usage, the corresponding preset threshold may be 5%. Specifically, the highest TaskManagerCPU usage rate may be obtained according to the obtained highest TaskManagerCPU usage, and the highest TaskManagerCPU usage rate may be compared with 5%. If it is lower than 5%, resource usage details information of the highest TaskManagerCPU usage rate being low may be generated.

[0120] When the resource usage indicator is the usage of the task sending end buffer and the usage of the task receiving end buffer, generating the resource usage details information may include the following steps:

[0121] According to the usage of the task sending end buffer and the usage of the task receiving end buffer, the usage rate of the corresponding task sending end buffer and the usage rate of the task receiving end buffer are obtained respectively.

[0122] When the usage rate of the corresponding ExclusiveBuffer (private Buffer) is lower than 10% and the usage rate of the FloatingBuffer (shareable Buffer) is higher than 90%, the cause analysis information is generated that there may be data skew.

[0123] When the Buffer occupancy rates of the receiving end inPoolUsage (input capacity) and the sending end outPoolUsage (output capacity) of a Subtask (subtask) are both less than 10%, it indicates normal.

[0124] When the inPoolUsageBuffer occupancy rate of a Subtask's receiving end is higher than 90% and the outPoolUsageBuffer occupancy rate of the sending end is lower than 10%, the cause analysis information is generated as the back pressure is transmitted to the upstream, which may be the root cause of the back pressure.

[0125] When the buffer occupancy rates of inPoolUsage and outPoolUsage at the receiving end of a Subtask are both higher than 90%, the cause analysis information is generated as being under downstream back pressure.

[0126] When the resource usage indicators are committedOffsets and currentOffsets, if the current committed offset position of each partition successfully submitted to Kafka is much lower than the offset position in the current read partition, it means that there is a backlog. In this case, the reason analysis information such as whether the resource configuration is reasonable and whether the business data flow increases is generated.

[0127] Through this implementation method, the present application will generate resource usage detail information based on the actual usage of each resource usage indicator, and send the detail information to the terminal for display so that relevant personnel can analyze the data and improve the flexibility of the solution.

[0128] In one embodiment, the above method may further include:

[0129] Receive a configuration request for a mapping relationship between a total resource usage score and an adjustment strategy;

[0130] According to the newly added adjustment policy in the configuration request and the total resource usage score corresponding to the newly added adjustment policy, a mapping relationship between the newly added adjustment policy and the total resource usage score corresponding to the newly added adjustment policy is created.

[0131] In this application, when a new adjustment strategy is needed, a configuration request for the mapping relationship between the total resource usage score and the adjustment strategy can be sent through the terminal. The configuration request carries the new adjustment strategy and the total resource usage score corresponding to the new adjustment strategy. According to the configuration request, a mapping relationship between the total resource usage score corresponding to the new adjustment strategy is created, and the mapping relationship is stored in a preset database. The adjustment strategy of this application supports modification and addition, which improves the flexibility of the solution.

[0132] In one embodiment, Figure 5 As shown, a resource adjustment device for Flink task processing is provided, including: an acquisition module 11, a generation module 12, a processing module 13 and an adjustment module 14, wherein:

[0133] The acquisition module 11 is used to obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task;

[0134] A generation module 12 is used to generate a total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator;

[0135] The processing module 13 is used to obtain the resource adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy;

[0136] The adjustment module 14 is used to adjust the resource allocation of the current Flink task according to the resource adjustment strategy of the current Flink task.

[0137] In one embodiment, the above-mentioned indicator parameters include a maximum value and a maximum indicator score. The above-mentioned generation module 12 can generate a usage ratio of the resource usage indicators of each first factor according to the actual usage and the maximum value of the resource usage indicators of each first factor when the evaluation factor to which the resource usage indicator belongs is the first factor, and generate a first score according to the usage ratio of the resource usage indicators of each first factor and the highest indicator score. When the evaluation factor to which the resource usage indicator belongs is the second factor, the resource usage indicator includes the first indicator and the second indicator, and generates a consumption ratio according to the actual usage of the first indicator and the actual usage of the second indicator. A second score is generated according to the consumption ratio and the highest indicator score of the resource usage indicator of the second factor, and a total resource usage score of the current Flink task is generated according to the second score and each first score.

[0138] In one embodiment, the above-mentioned processing module 13 can obtain the adjustment strategy that matches the interval to which the total resource usage score of the current Flink task belongs based on the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, and obtain the adjustment strategy of the current Flink task.

[0139] In one of the embodiments, the adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted, and the adjustment module 14 can adjust the resource allocation amount corresponding to each resource usage indicator of the current Flink task according to the adjustment value corresponding to the resource usage indicator to be adjusted.

[0140] In one of the embodiments, the above-mentioned device also includes a receiving module (not shown in the figure), which can execute the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task when receiving a resource adjustment request sent by the terminal, or, when monitoring that the current Flink task starts to run, execute the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task.

[0141] In one embodiment, the above-mentioned device also includes a comparison module (not shown in the figure), which can compare the actual usage corresponding to each resource usage indicator with a preset threshold to obtain a comparison result, and generate resource usage detail information corresponding to each resource usage indicator based on the comparison result, and send the resource usage detail information to the terminal for display.

[0142] In one embodiment, the above-mentioned device also includes a new module (not shown in the figure), which can receive a configuration request for a mapping relationship between a total resource usage score and an adjustment strategy, and create a mapping relationship between a new adjustment strategy and the total resource usage score corresponding to the new adjustment strategy based on the new adjustment strategy in the configuration request and the total resource usage score corresponding to the new adjustment strategy.

[0143] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the operation data of smart home devices. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a resource allocation method for compiling a virtual machine is implemented.

[0144] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task; generating a total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator; obtaining an adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and a mapping relationship between the pre-configured total resource usage score and the adjustment strategy;

[0145] Adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0146] In one embodiment, the above-mentioned indicator parameters include a maximum value and a maximum indicator score. When the processor executes a computer program to implement the above-mentioned step of generating a total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator, the following steps are specifically implemented:

[0147] When the evaluation factor to which the resource usage indicator belongs is the first factor, the usage ratio of the resource usage indicator of each first factor is generated according to the actual usage and the maximum value of the resource usage indicator of each first factor;

[0148] Generate a first score based on the usage ratio of the resource usage indicators of each first factor and the highest score of the indicator;

[0149] When the evaluation factor to which the resource usage indicator belongs is the second factor, the resource usage indicator includes the first indicator and the second indicator, and the consumption proportion is generated according to the actual usage of the first indicator and the actual usage of the second indicator;

[0150] Generate a second score based on the consumption proportion and the highest score of the resource use indicator of the second factor;

[0151] A total resource usage score of the current Flink task is generated based on the second score and each first score.

[0152] In one embodiment, when the processor executes the computer program to implement the above-mentioned mapping relationship between the total resource usage score of the current Flink task and the pre-configured total resource usage score and the adjustment strategy, and obtains the adjustment strategy step of the current Flink task, the following steps are specifically implemented:

[0153] According to the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy matching the interval to which the total resource usage score of the current Flink task belongs is obtained, and the adjustment strategy of the current Flink task is obtained.

[0154] In one of the embodiments, the adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted. When the processor executes the computer program to implement the step of adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task, the following steps are specifically implemented:

[0155] Adjust the resource allocation corresponding to each resource usage indicator of the current Flink task according to the adjustment value corresponding to the resource usage indicator to be adjusted.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] When receiving the resource adjustment request sent by the terminal, the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is executed;

[0158] Alternatively, when it is monitored that the current Flink task starts to run, a step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is performed.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] Compare the actual usage corresponding to each resource usage indicator with the preset threshold to obtain a comparison result;

[0161] Generate resource usage details information corresponding to each resource usage indicator according to the comparison results;

[0162] Send resource usage details to the terminal for display.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] Receive a configuration request for a mapping relationship between a total resource usage score and an adjustment strategy;

[0165] According to the newly added adjustment policy in the configuration request and the total resource usage score corresponding to the newly added adjustment policy, a mapping relationship between the newly added adjustment policy and the total resource usage score corresponding to the newly added adjustment policy is created.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task; generating a total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator; obtaining an adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and a mapping relationship between the pre-configured total resource usage score and the adjustment strategy; and adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

[0167] In one embodiment, the above-mentioned indicator parameters include a maximum value and a maximum indicator score. When the computer program is executed by the processor to implement the above-mentioned step of generating a total resource usage score of the current Flink task according to the actual usage of each resource usage indicator and the pre-configured indicator parameters of each resource usage indicator, the following steps are specifically implemented:

[0168] When the evaluation factor to which the resource usage indicator belongs is the first factor, the usage ratio of the resource usage indicator of each first factor is generated according to the actual usage and the maximum value of the resource usage indicator of each first factor;

[0169] Generate a first score based on the usage ratio of the resource usage indicators of each first factor and the highest score of the indicator;

[0170] When the evaluation factor to which the resource usage indicator belongs is the second factor, the resource usage indicator includes the first indicator and the second indicator, and the consumption proportion is generated according to the actual usage of the first indicator and the actual usage of the second indicator;

[0171] Generate a second score based on the consumption proportion and the highest score of the resource use indicator of the second factor;

[0172] A total resource usage score of the current Flink task is generated based on the second score and each first score.

[0173] In one embodiment, when the computer program is executed by the processor to implement the above-mentioned mapping relationship between the total resource usage score of the current Flink task and the pre-configured total resource usage score and the adjustment strategy, and obtain the adjustment strategy step of the current Flink task, the following steps are specifically implemented:

[0174] According to the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, the adjustment strategy matching the interval to which the total resource usage score of the current Flink task belongs is obtained, and the adjustment strategy of the current Flink task is obtained.

[0175] In one of the embodiments, the adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted. When the computer program is executed by the processor to implement the above-mentioned step of adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task, the following steps are specifically implemented:

[0176] Adjust the resource allocation corresponding to each resource usage indicator of the current Flink task according to the adjustment value corresponding to the resource usage indicator to be adjusted.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0178] When receiving the resource adjustment request sent by the terminal, the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is executed;

[0179] Alternatively, when it is monitored that the current Flink task starts to run, a step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task is performed.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0181] Compare the actual usage corresponding to each resource usage indicator with the preset threshold to obtain a comparison result;

[0182] Generate resource usage details information corresponding to each resource usage indicator according to the comparison results;

[0183] Send resource usage details to the terminal for display.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented:

[0185] Receive a configuration request for a mapping relationship between a total resource usage score and an adjustment strategy;

[0186] According to the newly added adjustment policy in the configuration request and the total resource usage score corresponding to the newly added adjustment policy, a mapping relationship between the newly added adjustment policy and the total resource usage score corresponding to the newly added adjustment policy is created.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0188] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A resource adjustment method for Flink task processing, the method comprising: Get the actual usage of each preset resource usage indicator during the execution of the current Flink task; When the evaluation factor to which the resource usage indicator belongs is the first factor, the usage ratio of the resource usage indicator of each first factor is generated according to the actual usage and the maximum value of the resource usage indicator of each first factor, the first factor includes memory, CPU, Full GC, back pressure and data skew, the resource usage indicator of Full GC includes the number of Full GC times and the Full GC time, and the resource usage indicators corresponding to the back pressure and data skew include the usage of the task sending end Buffer and the usage of the task receiving end Buffer; Generate a first score according to the usage proportion of the resource usage indicator of each of the first factors and the highest score of the indicator; When the evaluation factor to which the resource usage indicator belongs is Kafka consumption capacity, the resource usage indicator includes committedOffsets and currentOffsets, and the consumption proportion is generated according to the actual usage of committedOffsets and currentOffsets; Generate a second score according to the consumption proportion and the highest score of the corresponding indicator of the Kafka consumption capacity; Generate a total resource usage score of the current Flink task according to the second score and the sum of the first scores; According to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy, the adjustment strategy of the current Flink task is obtained, wherein the adjustment strategy includes resource usage detail information, and the generation process of the resource usage detail information includes the following steps: Determine whether there is data skew based on the usage of the task sender's buffer and the task receiver's buffer. Determine the back pressure situation based on the buffer occupancy of the task receiving end's input capacity and the task sending end's output capacity; Determine whether there is a backlog based on committedOffsets and currentOffsets; The resource allocation of the current Flink task is adjusted according to the adjustment strategy of the current Flink task.

2. The method according to claim 1, characterized in that The obtaining the adjustment strategy of the current Flink task according to the total resource usage score of the current Flink task and the mapping relationship between the pre-configured total resource usage score and the adjustment strategy includes: According to the mapping relationship between the interval to which the total resource usage score belongs and the adjustment strategy in the mapping relationship between the total resource usage score and the adjustment strategy, an adjustment strategy matching the interval to which the total resource usage score of the current Flink task belongs is obtained to obtain the adjustment strategy of the current Flink task.

3. The method according to claim 1, characterized in that: The adjustment strategy of the current Flink task includes an adjustment value corresponding to the resource usage indicator to be adjusted, and the adjusting the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task includes: The resource allocation amount corresponding to each resource usage indicator of the current Flink task is adjusted according to the adjustment value corresponding to the resource usage indicator to be adjusted.

4. The method according to claim 1, characterized in that The method further comprises: When receiving the resource adjustment request sent by the terminal, executing the step of obtaining the actual usage of each preset resource usage indicator during the execution of the current Flink task; Alternatively, when it is monitored that the current Flink task starts to run, the step of obtaining the actual usage amount of each preset resource usage indicator during the execution of the current Flink task is performed.

5. The method according to claim 4, characterized in that The method further comprises: Compare the actual usage corresponding to each resource usage indicator with the preset threshold to obtain a comparison result; Generate resource usage details information corresponding to each resource usage indicator according to the comparison result; The resource usage details information is sent to the terminal for display.

6. The method according to claim 1, characterized in that The method further comprises: Receiving a configuration request for a mapping relationship between the total resource usage score and the adjustment strategy; According to the newly added adjustment policy in the configuration request and the total resource usage score corresponding to the newly added adjustment policy, a mapping relationship between the newly added adjustment policy and the total resource usage score corresponding to the newly added adjustment policy is created.

7. A resource adjustment device for Flink task processing, the device comprising: The acquisition module is used to obtain the actual usage of each preset resource usage indicator during the execution of the current Flink task; A generation module, for generating a usage ratio of the resource usage indicators of each of the first factors according to the actual usage and maximum value of the resource usage indicators of each of the first factors when the evaluation factor to which the resource usage indicator belongs is the first factor, the first factors including memory, CPU, Full GC, back pressure and data skew, the resource usage indicators of Full GC including the number of Full GC and Full GC time, the resource usage indicators corresponding to the back pressure and data skew including the usage of the task sending end Buffer and the usage of the task receiving end Buffer, generating a first score according to the usage ratio of the resource usage indicators of each of the first factors and the highest score of the indicators, when the evaluation factor to which the resource usage indicator belongs is Kafka consumption capacity, the resource usage indicators including committedOffsets and currentOffsets, generating a consumption ratio according to the actual usage of committedOffsets and currentOffsets, generating a second score according to the consumption ratio and the highest score of the indicator corresponding to the Kafka consumption capacity, and generating a total resource usage score of the current Flink task according to the sum of the second score and the first scores; A processing module, configured to obtain an adjustment strategy for the current Flink task according to a total resource usage score of the current Flink task and a mapping relationship between a pre-configured total resource usage score and an adjustment strategy, wherein the adjustment strategy includes resource usage detail information, and a generation process of the resource usage detail information includes the following steps: determining whether data skew exists according to a usage of a buffer at a task sending end and a usage of a buffer at a task receiving end, determining a back pressure situation according to a buffer occupancy rate of an input capacity of a task receiving end and an output capacity of a task sending end, and determining whether there is a backlog according to committedOffsets and currentOffsets; The adjustment module is used to adjust the resource allocation of the current Flink task according to the adjustment strategy of the current Flink task.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images