Resource adjustment method, computer equipment and storage medium

Abstract

The embodiment of the present invention provides a resource adjustment method, a computer device and a storage medium. In certain implementation processes, the Worker monitors the resource utilization of its own physical resources, so that the actual utilization of resources in the physical resource pool can be determined based on its own physical resource utilization, thereby understanding its own actual burden, and allowing the Master to adjust the total amount of virtual resources for the Worker based on the Worker's actual burden, thereby establishing a connection between virtual resources and actual physical resources, avoiding the mismatch between the remaining virtual resources on the Worker side and the remaining physical resources, resulting in unreasonable resource utilization. The resource adjustment scheme of the embodiment of the present invention is conducive to increasing the throughput of tasks on the Worker side, improving the task execution performance of the Worker, and achieving optimal resource configuration.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of computers, and specifically, relate to, but are not limited to, a resource adjustment method, a computer device, and a storage medium. Background Art

[0002] Since the actual resource consumption of Spark tasks is affected by many factors such as the Stage, data volume, computational complexity, disk and network performance, etc., when the Spark Worker (work node) task burden is not heavy, the Core (computing core) and Memory (memory) resources applied for and obtained by the task at startup are not always or simultaneously fully occupied, which easily leads to resource waste; in other situations, although the existing tasks do not completely occupy all the Cores and Memory of the Worker, the actual burden of the Worker is already very heavy. At this time, if the remaining Core and Memory resources continue to be allocated to new tasks, it will increase the system load and affect the operation of the Spark Executor and the normal operation of the operating system. Summary of the invention

[0003] The resource adjustment method, computer device and storage medium provided by the embodiments of the present invention mainly solve the technical problem that Spark's resource management and task scheduling cannot perceive the actual burden of Spark Worker, which easily leads to unreasonable use of resources.

[0004] To solve the above technical problems, an embodiment of the present invention provides a resource adjustment method, which is applied to the Worker side in a Spark cluster. The resource adjustment method includes:

[0005] Monitor the resource utilization of the physical resources of this Worker in the current monitoring cycle;

[0006] Evaluate the utilization of physical resources during the monitoring period based on resource utilization and resource utilization critical interval;

[0007] If the evaluation result indicates that the utilization of physical resources during the monitoring period is unreasonable, a resource adjustment request is generated;

[0008] Send a resource adjustment request to the Master to request the Master to adjust the total amount of virtual resources of the Worker according to the resource adjustment request.

[0009] The embodiment of the present invention further provides a resource adjustment method, which is applied to the Master side in a Spark cluster. The resource management method includes:

[0010] Receive resource adjustment requests sent by Workers. Resource adjustment requests are generated based on the critical interval of resource utilization and the resource utilization of the Worker's physical resources in the most recent monitoring cycle, indicating that the utilization of the Worker's physical resources in the monitoring cycle is unreasonable.

[0011] Adjust the total amount of virtual resources of the Worker based on the resource adjustment request.

[0012] An embodiment of the present invention further provides a computer device, characterized in that the computer device includes a processor, a memory and a communication bus;

[0013] The communication bus is used to realize the connection and communication between the processor and the memory;

[0014] The processor is used to execute one or more computer programs stored in the memory to implement the steps on the Worker side of the aforementioned resource adjustment method; or the processor is used to execute one or more computer programs stored in the memory to implement the steps on the Master side of the aforementioned resource adjustment method.

[0015] An embodiment of the present invention also provides a storage medium, which stores at least one of a first resource adjustment program and a second resource adjustment program, wherein the first resource adjustment program can be executed by one or more processors to implement the steps on the Worker side of the aforementioned resource adjustment method; the second resource adjustment program can be executed by one or more processors to implement the steps on the Master side of the aforementioned resource adjustment method.

[0016] The beneficial effects of the present invention are:

[0017] The resource adjustment method, computer device and storage medium provided by the embodiment of the present invention, the Worker in the Spark cluster monitors the resource utilization of its own physical resources in the current monitoring cycle, and then evaluates the utilization of its own physical resources in the monitoring cycle according to the resource utilization and the critical interval of the resource utilization. If the evaluation result indicates that the utilization of the physical resources in the monitoring cycle is unreasonable, the Worker generates a resource adjustment request and sends the resource adjustment request to the Master. After receiving the resource adjustment request from the Worker, the Master adjusts the total amount of virtual resources of the Worker according to the resource adjustment request. In the embodiment of the present invention, the Worker monitors the resource utilization of its own physical resources, so that the actual utilization of resources in the physical resource pool can be determined based on its own physical resource utilization, so as to understand its actual burden, and let the Master adjust the total amount of virtual resources for the Worker based on the actual burden of the Worker, so as to establish a connection between virtual resources and actual physical resources, and avoid the mismatch between the remaining virtual resources on the Worker side and the remaining physical resources, resulting in unreasonable utilization of resources. Through the resource adjustment scheme of the embodiment of the present invention, it is conducive to increasing the throughput of tasks on the Worker side, improving the task execution performance of the Worker, and realizing the optimal configuration of resources.

[0018] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least some of the beneficial effects are obvious from the description of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An interactive flow chart of the resource adjustment method provided in Embodiment 1 of the present invention;

[0020] Figure 2 A flow chart of Worker monitoring resource utilization provided in Embodiment 1 of the present invention;

[0021] Figure 3 A schematic diagram of the latest N monitoring cycles shown in the first embodiment of the present invention;

[0022] Figure 4 This is a processing flow chart after the Master receives an increase request shown in the first embodiment of the present invention;

[0023] Figure 5 This is a processing flow chart after the Master receives a downgrade request shown in the first embodiment of the present invention;

[0024] Figure 6 A flow chart of a resource adjustment solution implemented on the Worker side provided in the second embodiment of the present invention;

[0025] Figure 7 A flow chart of a resource adjustment solution implemented by the Master side provided in the second embodiment of the present invention;

[0026] Figure 8 A schematic diagram of a hardware structure of a computer device provided in Embodiment 3 of the present invention;

[0027] Fig. 9 A schematic diagram of a Spark cluster provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention through specific implementation methods combined with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] Embodiment 1:

[0030] As one of the most popular big data computing frameworks, Apache Spark has great advantages in performance, stability, and ease of use. However, in terms of resource management and task scheduling, it cannot dynamically adjust computing resources according to the real-time load of the system, which easily leads to the problem of mismatch between computing resources and physical resources. In a production environment with limited resources but a large number of batch processing tasks, the following two types of problems often occur:

[0031] 1) All configured computing resources (Core, Memory) are allocated, but the utilization of physical resources such as CPU and memory is very low. However, the task queue is backlogged due to the exhaustion of computing resources and cannot be scheduled in real time, resulting in resource waste;

[0032] 2) When there are still some configured computing resources left, but the overall system CPU, memory, IO (Input Output) and other physical resource utilization is too high due to the occupation of JVM (Java Virtual Machine) GC (Garbage Collection) and other processes, tasks will continue to be scheduled and computing resources will be allocated to new tasks, which will increase the system load and affect the operation of Spark Executor and the normal operation of the operating system.

[0033] Spark's currently commonly used resource scheduling methods, such as Standalone (singleton mode) and Yarn (Yet Another Resource Negotiator), are aware of the problems in resource scheduling and have implemented some automatic resource adjustment solutions. However, they mainly evaluate whether to increase or decrease the number of executors based on the available resource margin, data volume, time consumption, etc., and do not perceive the utilization of physical resources, so they cannot solve the above two types of problems well.

[0034] Since the number of Spark Cores only limits the number of main computing threads and is not bound to the physical resource CPU, some common scheduling components have proposed the concept of virtual resources, such as CPU-vcores (virtual CPU cores) proposed in the Yarn resource scheduling method. It can appropriately increase or decrease the total number of available cluster resources at the beginning according to the performance of the physical CPU, but the cluster does not have the ability to dynamically adjust during operation. Therefore, there is still unreasonable resource utilization in this solution. In order to solve this problem, this embodiment provides a resource adjustment method, please refer to Figure 1 The following is a schematic diagram of the interaction process between Worker and Master in this method:

[0035] S102: The Worker monitors the resource utilization of its own physical resources in the current monitoring period.

[0036] In this embodiment, the Worker can monitor the resource utilization of its own physical resources in the current monitoring cycle, so as to understand the actual occupation of its own physical resources according to the resource utilization obtained by monitoring. In some examples of this embodiment, the Worker can start monitoring its own resource utilization when necessary, for example, only when the number of tasks being executed reaches a predetermined number, the monitoring of the resource utilization of physical resources is started, or only when the number of remaining virtual resources is small, the monitoring of the resource utilization of physical resources is started. In some other examples of this embodiment, the Worker can also monitor the resource utilization of its own physical resources in real time and continuously.

[0037] It is understandable that, under normal circumstances, the length of the monitoring cycle used by the Worker to monitor the utilization of physical resources is consistent. For example, in some examples of this embodiment, the length of each monitoring cycle is 10s. However, this embodiment does not exclude the existence of some monitoring cycles that are not completely the same. For example, in some examples, when there are more virtual resources remaining on the Worker side, the monitoring cycle can be relatively long, but as the number of remaining virtual resources continues to decrease, the monitoring cycle on the Worker side becomes shorter and shorter. In this embodiment, after the Worker monitors for a monitoring cycle, the Master may adjust the total amount of virtual resources on the Worker side based on the monitoring results. Therefore, the longer the monitoring cycle, the longer the opportunity interval for the total amount of virtual resources on the Worker side to be adjusted. On the contrary, the shorter the monitoring cycle, the shorter the opportunity interval for the total amount of virtual resources on the Worker side to be adjusted. In this way, if there is unreasonable resource utilization on the Worker side, it can be adjusted in time.

[0038] In some examples of this embodiment, the physical resources on the Worker side may include but are not limited to CPU, IO, and memory. Considering that the CPU can more accurately reflect the utilization of the physical resources on the Worker side, in some examples of this embodiment, the utilization of the CPU on the Worker side can be directly monitored as a resource utilization indicator for monitoring. In other examples of this embodiment, it is also possible to consider monitoring the utilization of memory as a resource utilization indicator for monitoring. In some examples of this embodiment, there may be a situation where the CPU is sufficient but the IO resources are limited. In these examples, it is possible to choose to monitor the utilization of IO as a resource utilization indicator for monitoring. Of course, in order to fully and comprehensively understand the occupancy of physical resources on the Worker, in some examples of this embodiment, it is possible to choose to monitor multiple resource utilization indicators at the same time, so as to combine the monitoring results of these resource utilization indicators to evaluate the utilization of the Worker's physical resources. For example, in one example of this embodiment, the Worker monitors the utilization of its CPU, IO, and memory at the same time. Of course, in some other examples, the Worker can also choose to monitor only two of the resource utilization indicators.

[0039] In some examples of this embodiment, the worker can follow Figure 2 Shown is the flow chart to monitor the resource utilization of physical resources:

[0040] S202: The Worker collects its own physical resource statistics in the current monitoring cycle.

[0041] The Worker can periodically collect physical resource statistics from the operating system according to the monitoring cycle. It should be understood that the physical resource statistics collected by the Worker should at least include the statistics corresponding to the resource utilization indicators to be monitored. For example, in an example of the present embodiment, the resource utilization indicators to be monitored by the Worker include CPU utilization and IO utilization, and the physical resource statistics collected should at least include CPU statistics and IO statistics. Of course, it can be understood by those skilled in the art that although the resource utilization indicators monitored by the Worker only include CPU utilization and IO utilization, in some examples, the physical resource statistics collected by the Worker can also include memory statistics. However, in more examples, in order to reduce the burden of the Worker when monitoring the resource utilization of physical resources and avoid unnecessary data collection by the Worker, only the statistics corresponding to the resource utilization indicators that the Worker needs to monitor can be collected, and other irrelevant statistics do not need to be collected.

[0042] S204: The Worker determines the real-time utilization of the physical resources in the monitoring period according to the physical resource statistical data.

[0043] It should be understood that when the Worker completes the collection of physical resource statistics for a monitoring cycle, the monitoring cycle also ends. Therefore, when the Worker completes data collection and starts to use these data to determine resource utilization, the original "current monitoring cycle" has become the historical monitoring cycle closest to the current moment. In the subsequent introduction, for the sake of convenience, this monitoring cycle is called the "most recent monitoring cycle."

[0044] Based on the statistical data collected by the Worker, the real-time utilization of the physical resources that the Worker requires to monitor in the most recent monitoring cycle can be determined. The calculation method refers to the calculation of the operating system Top. For example, if the resource utilization indicator required to be monitored includes the CPU utilization, the Worker can determine the duration t1 that the CPU is in use in the most recent monitoring cycle, and then calculate the ratio between t1 and the total duration t of the most recent monitoring cycle as the real-time utilization of the CPU in the current monitoring cycle. The calculation method for IO real-time utilization is the same.

[0045] S206: The Worker determines the average of the instantaneous utilization rates of each monitoring period in the most recent M monitoring periods as the resource utilization rate of the physical resource in the most recent monitoring period.

[0046] In theory, the instant utilization calculated by the Worker can characterize the utilization of its physical resources in the most recent monitoring cycle. Therefore, in some examples of this embodiment, the value of M is 1, and the Worker can directly evaluate whether its resource utilization is reasonable based on the instant utilization determined by itself. However, after all, a monitoring cycle is not long, and the physical resource utilization of the Worker may occasionally change suddenly. Therefore, if the resource utilization is evaluated only based on the instant utilization, and the total number of virtual resources of the Worker is adjusted according to the evaluation results, the virtual resources of the Worker may need to be adjusted after the monitoring cycle where the mutation occurs. However, after the mutation ends and the resource utilization level of the Worker returns to normal, the virtual resources of the Worker need to be adjusted again. After the virtual resource adjustment, resource scheduling will be triggered, and frequent resource scheduling means continuous adjustment and release of resources, which will seriously affect the performance of Spark task operation.

[0047] Therefore, in some examples of this embodiment, the value of M is an integer greater than 1, that is, the Worker does not evaluate its physical resource utilization based solely on the instant utilization, but evaluates based on the instant utilization of at least two or even more recent monitoring cycles, thereby smoothing out sudden changes in resource utilization. In some examples of this embodiment, M is an integer greater than or equal to 2, and the Worker calculates the average of the instant utilization of the most recent M historical monitoring cycles, and uses the average as the resource utilization of the most recent monitoring cycle.

[0048] In some examples of this embodiment, if the physical resources to be monitored include CPU, the resource utilization may include average CPU utilization; if the physical resources to be monitored include IO, the resource utilization may include average IO utilization.

[0049] S104: The Worker evaluates the utilization of the physical resources in the most recent monitoring period according to the resource utilization and the critical range of the resource utilization.

[0050] In this embodiment, the Worker will use the resource utilization critical interval as an evaluation criterion to evaluate the physical resource utilization of the most recent monitoring period determined by itself. It can be understood that the resource utilization critical interval, as an interval, will include an upper limit value and a lower limit value. In this embodiment, the resource utilization critical interval includes the resource utilization upper limit and the resource utilization lower limit. For example, if the resource utilization includes the average CPU utilization, the resource utilization critical interval includes the average CPU utilization critical interval, the resource utilization upper limit includes the average CPU utilization upper limit, and the resource utilization lower limit includes the average CPU utilization lower limit. If the resource utilization includes the average IO utilization, the resource utilization critical interval includes the average IO utilization critical interval, the resource utilization upper limit includes the average IO utilization upper limit, and the resource utilization lower limit includes the average IO utilization lower limit.

[0051] The Worker can compare the resource utilization calculated by itself with the upper and lower limits of resource utilization: if the resource utilization corresponding to the most recent monitoring period is between the upper and lower limits of resource utilization (including the upper and lower limits of resource utilization themselves in some examples), then it can be considered that the utilization of the Worker's physical resources in the most recent monitoring period is good. In some examples of this embodiment, as long as the resource utilization corresponding to the most recent monitoring period is outside the critical interval of resource utilization, the Worker will determine that the utilization of its physical resources in the most recent monitoring period is unreasonable, that is, after determining that the comparison result is any of the following, the Worker determines that the utilization of its physical resources in the most recent monitoring period is unreasonable:

[0052] The first one is that the resource utilization in the most recent monitoring period is higher than the resource utilization upper limit;

[0053] The second type is that the resource utilization in the most recent monitoring period is lower than the lower limit of the resource utilization.

[0054] In some examples of this embodiment, when the comparison result of the resource utilization rate in the most recent monitoring period with the critical interval of resource utilization is the second case, the Worker needs to further determine whether the same comparison result has appeared continuously in the most recent N monitoring periods before determining that the physical resource utilization is unreasonable. In some examples of this embodiment, the value of N is an integer greater than or equal to 2. For example, see Figure 3, assuming that the value of N is 3, then when the Worker determines at time t0 that the comparison result of the resource utilization rate in the most recent monitoring cycle and the critical interval of resource utilization is the second type mentioned above, it is also necessary to determine whether the comparison results in the two monitoring cycles (a1 and a2) before the "most recent monitoring" a0 are also the second type mentioned above. If so, the Worker can determine that its physical resource utilization is unreasonable. Assuming that the value of N is 6, then after the Worker determines that the comparison result of the resource utilization rate in the most recent monitoring cycle and the critical interval of resource utilization is the second type mentioned above, it is also necessary to determine whether the comparison results in the five monitoring cycles before the "most recent monitoring" are also the second type mentioned above. If so, the Worker can determine that its physical resource utilization is unreasonable.

[0055] In some examples of this embodiment, the resource utilization monitored by the Worker includes at least one of the average CPU utilization and the average IO utilization. When it is determined that the comparison result of the resource utilization in the most recent monitoring period with the critical interval of resource utilization is the second case described above, the Worker will further evaluate the physical resource utilization in combination with the memory utilization before determining that the physical resource utilization is unreasonable: For example, in one example of this embodiment, the resource utilization monitored by the Worker includes both the average CPU utilization and the average IO utilization. When the Worker determines that the average CPU utilization in the most recent monitoring period is lower than the lower limit of the average CPU utilization, and the average IO utilization is lower than the lower limit of the average IO utilization, the Worker will judge whether the idle rate of its own memory is lower than the preset Free memory (free memory) safety red line, and determine whether the available rate of its own memory is lower than the Available memory (available memory) safety red line. If the Worker determines that the idle rate of its own memory is not lower than the Free memory safety red line, and the available rate is not lower than the Available memory safety red line, it can be determined that the utilization of its own physical resources in the most recent monitoring period is low and the utilization is unreasonable.

[0056] It should be noted that in some examples, the Worker may also use only one of the Available memory safety red line and the Free memory safety red line to evaluate its own memory utilization, but in this embodiment, the Free Memory safety red line and the Available Memory safety red line are set at the same time, which reduces the frequency of the operating system reclaiming the Cache (cache memory) and using the Swap (swap area) partition.

[0057] Optionally, in an example of this embodiment, assuming that the values ​​of M and N are both 3, the resource utilization monitored by the Worker at the beginning includes the average CPU utilization and the average IO utilization: when the Worker process starts, the scheduled monitoring task is started, and the Worker begins to periodically collect physical resource statistics from the operating system, and calculates the instantaneous CPU utilization and the instantaneous IO utilization within the monitoring period based on the collected physical resource statistics, and then calculates the average CPU utilization and the average IO utilization within the last three periods. Compare the average CPU utilization with the upper limit of the average CPU utilization and the lower limit of the average CPU utilization, and also compare the average IO utilization with the upper limit of the average IO utilization and the lower limit of the average IO utilization:

[0058] If at least one of the average CPU utilization and the average IO utilization in the comparison result exceeds the upper limit of the corresponding critical interval, the Worker can determine that the utilization of its physical resources in the most recent monitoring period is too high and the utilization is unreasonable.

[0059] If the average CPU utilization and the average IO utilization in the comparison results are both lower than the corresponding critical interval lower limit, the Worker will further determine whether the same comparison result has appeared three times in a row, that is, whether the comparison results in the last three monitoring cycles are all of the second type. If so, the Worker will determine whether the idle rate of its own memory is lower than the preset Free memory safety red line, and whether the availability is lower than the Available memory safety red line. Only when these two judgment results are both negative, will the Worker determine that the utilization of its own physical resources in the most recent monitoring cycle is low and the utilization is unreasonable.

[0060] In the above example, when the Worker determines that its physical resources are used at a high level in the most recent monitoring cycle based on resource utilization, it will not further determine whether this situation has continued for a period of time (that is, when the comparison result is the first type, the Worker will not further determine whether this comparison result has appeared N times in a row), because the high utilization rate of the Worker's physical resources must mean that the Worker is heavily burdened, so it should be avoided in time to further increase its burden, otherwise it will affect the running performance of its Executor. However, when the Worker determines that its physical resources are used at a low level in the most recent monitoring cycle based on resource utilization, it needs to further determine whether this situation has continued for a period of time (that is, when the comparison result is the second type, the Worker will further determine whether this comparison result has appeared N times in a row), because the low utilization rate of the Worker's physical resources only affects the throughput of the Worker, at least it has no effect on the current running performance of its Executor, so the negative impact of this situation is smaller than the previous one. In addition, the low utilization rate of the Worker's physical resources may also be due to the fact that there are not many tasks that need to be executed on the Worker at present.

[0061] S106: If the evaluation result indicates that the utilization of physical resources in the most recent monitoring period is unreasonable, the Worker generates a resource adjustment request.

[0062] When the Worker uses the critical interval of resource utilization to evaluate the resource utilization of the most recent monitoring period and the evaluation result indicates that the utilization of physical resources in the most recent monitoring period is unreasonable, the Worker needs to feedback this situation to the Master so that the Master can adjust the total amount of virtual resources of the Worker in a timely manner to improve the current resource utilization. In some examples of this embodiment, the resource adjustment request sent by the Worker to the Master can at least indicate the direction in which it requests the Master to make resource adjustments, such as requesting to increase the total amount of virtual resources or requesting to reduce the total amount of virtual resources:

[0063] It is obvious that if the unreasonable utilization of physical resources in the Worker's recent monitoring cycle is because the resource utilization is higher than the critical interval of resource utilization, the Worker should request the Master to reduce the total amount of its virtual resources so that the Worker can start fewer Executors at the same time in the subsequent process. Therefore, in this case, the resource adjustment request generated by the Worker should be a Turn Down request. If the unreasonable utilization of physical resources in the Worker's recent monitoring cycle is because the resource utilization is lower than the critical interval of resource utilization, the Worker should request the Master to increase the total amount of its virtual resources so that the Worker can have the opportunity to start more Executors at the same time in the subsequent process to increase resource utilization and avoid resource waste. Therefore, the Worker can generate a Turn Up request as a resource adjustment request.

[0064] S108: The Worker sends a resource adjustment request to the Master.

[0065] After the Worker generates a resource adjustment request, it can send the resource adjustment request to the Master. In this embodiment, the specific communication method between the Worker and the Master is not limited, because when the Master and the Worker form a Spark cluster, they will achieve a communication connection, and the communication connection between the two can be wired or wireless. In some examples of this embodiment, the Worker sends the resource adjustment request to the Master through Rpc Endpoint (Remote Procedure Call uses a remote computer to send a request program, which can provide an endpoint mapping program and RPC service, and is one of the three major services of the Windows system).

[0066] S110: The Master adjusts the total amount of virtual resources of the Worker according to the resource adjustment request.

[0067] After receiving the resource adjustment request sent by the Worker, the Master can adjust the total amount of virtual resources of the Worker according to the resource adjustment request:

[0068] In some examples of this embodiment, if the resource adjustment request sent by the Worker is an increase request, the Master increases the total amount of virtual resources of the Worker; if the resource adjustment request sent by the Worker is a decrease request, the Master reduces the total amount of virtual resources of the Worker.

[0069] However, in some other examples of this embodiment, before the Master adjusts the total amount of its virtual resources according to the Worker's resource adjustment request, the Master will first determine whether the corresponding adjustment conditions are currently met. For example, if the resource adjustment request is an increase request, the Master can adjust the total amount of its virtual resources according to the Worker's resource adjustment request. Figure 4 The process shown is processed:

[0070] S402: The Master determines whether the sum of the Worker's cumulative virtual resource adjustment amount tc and the upward adjustment increment Δd exceeds the adjustment amount upper limit H.

[0071] If the judgment result is yes, then go to S404, otherwise go to S406. Wherein, tc is the difference between the total amount of virtual resources currently available to the Worker and the total amount of initial virtual resources at the start of the game, and Δd is the number of virtual resources expected to be added to the Worker this time.

[0072] S404: The Master increases the total amount of virtual resources of the Worker according to H-tc.

[0073] In some examples of this embodiment, H and Δd are in a multiple relationship, that is, H is an integer multiple of Δd. In this case, if the sum of tc and Δd exceeds H, it means that tc is equal to H, so the Master will maintain the current total amount of virtual resources of the Worker unchanged.

[0074] However, in some other examples, there is no exponential relationship between H and Δd. In this case, when the sum of tc and Δd exceeds H, there is still a certain difference between tc and H. Therefore, the total amount of virtual resources of the Worker still has a certain upward adjustment space. Therefore, the Master can increase the total amount of virtual resources of the Worker according to H-tc.

[0075] The reason why H is used in this embodiment to limit the unlimited increase in the total amount of Worker virtual resources is mainly to prevent the Master from giving the Worker too many virtual resources, causing the Worker to run too many executors at the same time. When the tasks executed by these executors enter the heavy stage one after another, the Worker may face excessive pressure.

[0076] S406: The Master increases the total amount of virtual resources of the Worker according to the upward adjustment increment Δd.

[0077] If the Master determines that the sum of tc and Δd is less than or equal to H, the Master can also increase the total amount of virtual resources of the Worker according to Δd. In this embodiment, the values ​​of H and Δd can be set by the system administrator before starting the monitoring task each time, or can be set in advance by the system designer and stored in the system.

[0078] If the resource adjustment request is a downward adjustment request, the Master can adjust the resource according to the following steps after receiving the resource adjustment request from the Worker: Figure 5 The process shown is processed:

[0079] S502: The Master determines whether the accumulated adjustment amount tc of the Worker virtual resource is greater than 0.

[0080] If the Master's judgment result is yes, then go to S504, otherwise go to S508.

[0081] As in the above example, tc still represents the difference between the total amount of virtual resources currently available to the Worker and the initial total amount of virtual resources at the start of the game.

[0082] S504: The Master determines the corresponding downward adjustment increment of the Worker according to the following formula:

[0083]

[0084] Wherein, x is the fallback multiple, and its value is a positive number greater than or equal to 1, and the larger its value is, the larger the absolute value of Δc is. It can be understood that if the value of x is large enough, the Master can quickly reduce the number of Worker virtual resources and reduce the pressure on the Worker side. In this embodiment, the value of x can also be set by the system administrator himself before starting each monitoring task, or set by the system designer and stored in the system.

[0085] S506: The Master reduces the total amount of virtual resources of the Worker according to the downward adjustment increment Δc.

[0086] It should be understood that if the value of tc is greater than 0, it means that the Master has increased the total amount of virtual resources for the Worker since the start of the game. Therefore, the Master can now reduce the amount of its virtual resources based on the current total amount of virtual resources of the Worker.

[0087] S508: The Master maintains the current total amount of virtual resources of the Worker unchanged.

[0088] Of course, those skilled in the art can understand that, in some other examples, even if the value of tc is not greater than 0, the Master can reduce the number of its virtual resources. However, because the value of the Worker's initial total amount of virtual resources is usually a more appropriate value set by the engineer according to the total amount of physical resources of the Worker, such an approach may cause the Worker to be unable to respond immediately when it suddenly receives some tasks. Instead, it needs to request to adjust the number of virtual resources again after monitoring the task backlog before scheduling these tasks.

[0089] Compared with the related schemes that cannot adjust the total amount of Worker virtual resources or the schemes that can only adjust the Worker virtual resources once at the start (such as Yarn resource scheduling method), the resource adjustment method provided by the embodiment of the present invention can achieve the effect of dynamically adjusting the total amount of Worker virtual resources through real-time monitoring of the utilization rate of Worker physical resources, so that Worker can fully utilize resources while maintaining resource utilization at a relatively ideal level. In the case of cooperating with Spark dynamic resource allocation, the throughput and task execution performance of Worker are taken into account.

[0090] At the same time, the Worker smooths the sudden changes in resource utilization by using the critical interval of resource utilization and the average of instant utilization, thus avoiding frequent adjustments. Moreover, when the resource utilization exceeds the upper limit of the critical interval of resource utilization, the Master can use a fast fallback mechanism to reduce the number of virtual resources of the Worker to quickly relieve the pressure on the Worker.

[0091] Embodiment 2:

[0092] In order to facilitate those skilled in the art to understand the advantages and details of the resource adjustment method provided in the embodiment of the present invention, this embodiment will continue to introduce the resource adjustment scheme in conjunction with a specific example. Figure 6 The flowchart of the resource adjustment method executed on the Worker side is shown:

[0093] S602: Obtain resource adjustment parameters.

[0094] In some examples of this embodiment, the resource adjustment parameters that the Worker needs to obtain include the critical interval of resource utilization and the monitoring cycle duration. In other examples, the Worker also needs to obtain the Free memory safety red line, the Available memory safety red line, and the values ​​of M and N. In this embodiment, the resource adjustment parameters obtained by the Worker include:

[0095] CPU utilization critical interval (%): [C1, C2]

[0096] IO utilization critical interval (%): [D1, D2]

[0097] Free memory safety red line (%): F1

[0098] Available memory safety red line (%): A1

[0099] Monitoring cycle (seconds): t

[0100] M: Max (10, 120 / t)

[0101] N:N0.

[0102] It is understandable that the minimum value of M is 10. In the value formula, both "10" and "120" can be set by the system administrator according to needs or by engineers according to experience.

[0103] S604: Start a resource utilization monitoring task.

[0104] S606: Collect physical resource statistics related to CPU utilization and IO utilization.

[0105] After starting the resource utilization monitoring task, the Worker will periodically collect physical resource statistical data according to t. For example, in an example of this embodiment, the Worker sets a timer with a duration of t. When the timer times out, the CPU and IO statistics are obtained from the operating system.

[0106] S608: Calculate the real-time utilization of the CPU and the real-time utilization of the IO.

[0107] Worker can calculate the real-time utilization of CPU and IO in this period by comparing the current collection results with the last collection results. The calculation method refers to the calculation of the operating system Top. The calculation results are stored in the CPU and IO cache queues Q1 and Q2 respectively. Both Q1 and Q2 have a fixed length M. If the queue is full, the head overflows.

[0108] S610: Calculate the average CPU utilization rate according to the real-time CPU utilization rates in the latest M monitoring cycles, and calculate the average IO utilization rate according to the real-time IO utilization rates in the latest M monitoring cycles.

[0109] After Q1 and Q2 are full, the Worker will calculate the average of the queues after the data in Q1 and Q2 is updated to obtain the average CPU utilization and the average IO utilization.

[0110] S612: Determine whether at least one of the average CPU utilization and the average IO utilization exceeds the corresponding upper limit.

[0111] If so, proceed to S614; otherwise, proceed to S616.

[0112] S614: Generate a downward adjustment request.

[0113] S616: Determine whether the average CPU utilization and the average IO utilization are both lower than corresponding lower limits.

[0114] If yes, proceed to S618, otherwise end the process.

[0115] S618: Determine whether the same comparison result has appeared N times in succession.

[0116] If yes, proceed to S620, otherwise the process ends.

[0117] Optionally, the Worker can set a counter count. Whenever the comparison result that the average CPU utilization and the average IO utilization are both lower than the corresponding lower limits appears once in a row, the value of the counter count is increased by 1 until the value of count reaches N0.

[0118] If, in a judgment before the value of count reaches N0, the comparison result is not that both the average CPU utilization and the average IO utilization are lower than the corresponding lower limits, the counter is cleared and starts counting again.

[0119] S620: Determine whether the idle rate of the own memory is lower than the Free memory safety red line, and whether the available rate is lower than the Available memory safety red line.

[0120] If yes, proceed to S622, otherwise the process ends.

[0121] S622: Generate an increase request.

[0122] It should be noted that after the production increase request is made, the counter should be cleared.

[0123] S624: Send the generated resource adjustment request to the Master.

[0124] After generating a resource adjustment request, the Worker sends it to the Master through RpcEndpoint.

[0125] If the Worker does not generate a resource adjustment request after the above judgment, it means that the Worker has made good use of its physical resources in recent period and does not need to adjust virtual resources for the time being.

[0126] Combine the following Figure 7 The flowchart shown introduces the process of executing the resource adjustment method on the Master side:

[0127] S702: The Master receives the resource adjustment request sent by the Worker.

[0128] Before executing the resource adjustment method, the master should first determine the resource adjustment parameters. In some examples of this embodiment, the resource adjustment parameters that the master needs to obtain include the adjustment upper limit H, the upward adjustment increment Δd, and the fallback multiple x.

[0129] In this embodiment, the resource adjustment parameters that the Master needs to obtain include:

[0130] Δd core , Core upward adjustment increment (number): n

[0131] Δd memory , memory adjustment increment (MB): m

[0132] Core adjustment limit: H

[0133] Retreat multiple: x.

[0134] Typically, the memory is adjusted upward by an increment of Δd memory Adjust the increment Δd with Core upward core It is a simple linear relationship, calculated as Δd memory =m·Δd core / n. In addition, it is generally recommended that H is an integer multiple of n.

[0135] S704: Determine whether the resource adjustment request is an upward adjustment request.

[0136] If the result of the judgment is yes, then S706 is executed, otherwise S710 is executed. It is undoubted that if the resource adjustment request is not an upward adjustment request, then it must be a downward adjustment request. It can be understood by those skilled in the art that the judgment process executed by the Master here can also be "judging whether the resource adjustment request is a downward adjustment request".

[0137] S706: Determine whether the sum of tc and Δd corresponding to the Worker exceeds H.

[0138] If the judgment result is yes, the process ends, otherwise S708 is executed. Ending the process means that the Master will maintain the current total amount of virtual resources of the Worker unchanged.

[0139] S708: Increase the total amount of virtual resources of the Worker according to the upward adjustment increment Δd.

[0140] S710: Determine whether tc corresponding to the Worker is greater than 0.

[0141] If the Master's judgment result is yes, then the process proceeds to S712, otherwise the process ends.

[0142] S712: Reduce the total amount of virtual resources of the Worker according to the downward adjustment increment Δc.

[0143] In this embodiment, the calculation method of Δc can refer to the introduction in the above-mentioned embodiment, which will not be repeated here.

[0144] S714: Perform scheduling according to the adjusted total amount of virtual resources.

[0145] After the Master completes the virtual resource adjustment, it applies the adjusted value to the Worker that currently reports the resource adjustment request, and updates the Worker's cumulative adjusted Core and Memory counts. Subsequently, the Master checks whether there are currently tasks being executed or waiting for resource allocation. If so, it triggers a schedule to increase or release computing resources in a timely manner.

[0146] Managing Worker resources according to the resource adjustment solution provided in this embodiment has the following advantages:

[0147] First, dynamically adjust the Worker virtual resources to make full use of cluster resources while maintaining resource utilization at an ideal level. In conjunction with Spark dynamic resource allocation, it can increase task throughput and improve task execution performance.

[0148] Second, the sudden changes in resource utilization are smoothed out by using the critical interval of resource utilization and the average of instant utilization, thus avoiding frequent adjustments.

[0149] Third, in addition to considering the average CPU utilization, this embodiment also supports monitoring the utilization rate of physical resources such as IO and memory, and comprehensively measures resource utilization through multiple indicators, which is conducive to comprehensively evaluating the resource utilization level of Workers.

[0150] Fourth, when considering the memory factor, the Free Memory safety redline and Available Memory safety redline are set to reduce the frequency of the operating system reclaiming the cache and using the swap partition. At the same time, in the Worker computing resource adjustment process, a one-time veto mechanism is adopted for the memory utilization (that is, there is no need to consider the memory utilization in other historical monitoring cycles). Considering that memory is inert and insufficient memory will lead to serious consequences such as GC and memory overflow, which directly affect the operation of tasks, the one-time veto mechanism greatly reduces the probability of problems.

[0151] Fifth, when the average resource utilization exceeds the upper limit of the critical interval, a rapid fallback mechanism is used to reduce the number of damping resources, which can quickly alleviate the pressure on the Worker caused by excessive resource utilization.

[0152] Embodiment three:

[0153] The present embodiment provides a storage medium, which includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). The storage medium includes but is not limited to RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by a computer.

[0154] The storage medium in this embodiment can be used to store one or more computer programs. For example, in some examples of this embodiment, it stores at least one of a first resource adjustment program and a second resource adjustment program. When the storage medium stores the first resource adjustment program, the program can be executed by the processor to implement the steps on the Worker side of the resource adjustment method in the first and second embodiments. When the storage medium stores the second resource adjustment program, the program can be executed by the processor to implement the steps on the Master side of the resource adjustment method in the first and second embodiments.

[0155] This embodiment also provides a computer device 80, see Figure 8 As shown, it includes a processor 82, a memory 84 and a communication bus 86, wherein:

[0156] The communication bus 86 is used to realize the connection and communication between the processor 82 and the memory 84; the memory 84 may be the storage medium storing at least one of the first resource adjustment program and the second resource adjustment program.

[0157] When the memory 84 stores the first resource adjustment program, the computer device 80 can act as a Worker, and its processor 82 can execute the first resource adjustment program stored in the memory 84 to implement the steps on the Worker side in the resource adjustment method in the first and second embodiments above:

[0158] The processor 82 monitors the resource utilization of the physical resources of the computer device 80 in the current monitoring cycle, and then evaluates the utilization of the physical resources in the monitoring cycle according to the resource utilization and the critical interval of the resource utilization. If the evaluation result indicates that the utilization of the physical resources in the monitoring cycle is unreasonable, a resource adjustment request is generated and sent to the master, requesting the master to adjust the total amount of virtual resources of the computer device 80 according to the resource adjustment request.

[0159] In this embodiment, the virtual resources include a computing core Core and a memory.

[0160] Optionally, the processor 82 may collect physical resource statistical data of the computer device 80 during the current monitoring cycle, and then determine the real-time utilization of the physical resources during the monitoring cycle based on the physical resource statistical data, and then calculate the average of the real-time utilization of each monitoring cycle in the most recent M monitoring cycles as the resource utilization of the physical resources during the monitoring cycle, where M is an integer greater than or equal to 1.

[0161] In an example of this embodiment, when the processor 82 evaluates the utilization of the physical resource in the monitoring period according to the resource utilization and the resource utilization critical interval, the resource utilization of the monitoring period may be compared with the resource utilization upper limit and the resource utilization lower limit of the resource utilization critical interval; after determining that the comparison result is any of the following, it is determined that the utilization of the physical resource in the monitoring period is unreasonable:

[0162] The first one is that the resource utilization rate in the monitoring period is higher than the resource utilization upper limit;

[0163] The second type is that the resource utilization during the monitoring period is lower than the lower limit of resource utilization.

[0164] In some examples of this embodiment, after determining that the comparison result is the second type and before determining that the utilization of physical resources within the monitoring period is unreasonable, the processor 82 will also determine that the same comparison result has appeared continuously in the last N monitoring periods, where N is an integer greater than or equal to 2.

[0165] Optionally, the resource utilization includes at least one of an average CPU utilization and an average input / output (IO) utilization. If the resource utilization includes an average CPU utilization, the critical interval of the resource utilization includes an average CPU utilization critical interval, the upper limit of the resource utilization includes an upper limit of the average CPU utilization, and the lower limit of the resource utilization includes an average CPU utilization lower limit; if the resource utilization includes an average IO utilization, the critical interval of the resource utilization includes an average IO utilization critical interval, the upper limit of the resource utilization includes an upper limit of the average IO utilization, and the lower limit of the resource utilization includes an average IO utilization lower limit.

[0166] In an example of this embodiment, resource utilization includes both CPU average utilization and IO average utilization; after the processor 82 determines that the comparison result is the second one and the same comparison result appears continuously in the last N monitoring cycles, before determining that the utilization of physical resources in the monitoring cycle is unreasonable, it will also evaluate the utilization of physical resources in the monitoring cycle in combination with the memory utilization of the computer device 80.

[0167] When the processor 82 evaluates the utilization of physical resources during the monitoring period in combination with the memory utilization of the computer device 80, if it is determined that the idle rate of the memory of the computer device 80 is not lower than the Free memory safety red line, and the available rate is not lower than the Available memory safety red line, it is determined that the utilization of physical resources during the monitoring period is unreasonable.

[0168] In some examples of this embodiment, the resource adjustment request can at least reflect the adjustment direction of the requested resource adjustment. When the processor 82 generates a resource adjustment request, if the resource utilization is lower than the critical interval of the resource utilization, then a request for increasing the total amount of virtual resources of the computer device 80 is generated; if the resource utilization is higher than the critical interval of the resource utilization, then a request for decreasing the total amount of virtual resources of the computer device 80 is generated.

[0169] When the memory 84 stores the second resource adjustment program, the computer device 80 can serve as a master, and its processor 82 can execute the second resource adjustment program stored in the memory 84 to implement the steps on the master side in the resource adjustment method in the first and second embodiments above:

[0170] The processor 82 receives the resource adjustment request sent by the Worker, and then adjusts the total amount of virtual resources of the Worker according to the resource adjustment request.

[0171] Optionally, the resource adjustment request can at least reflect the adjustment direction of the requested resource adjustment. When the processor 82 adjusts the total amount of virtual resources of the Worker according to the resource adjustment request, if the resource adjustment request is an upward adjustment request, the total amount of virtual resources of the corresponding Worker is increased; if the resource adjustment request is a downward adjustment request, the total amount of virtual resources of the corresponding Worker is reduced.

[0172] In some examples of this embodiment, before increasing the total amount of virtual resources of a Worker, the processor 82 will first determine that the sum of the Worker's cumulative adjustment amount tc and the upward adjustment increment Δd has not exceeded the adjustment amount upper limit H; where tc is the difference between the Worker's current total amount of virtual resources and the initial total amount of virtual resources at the start of the game; Δd ​​is the number of virtual resources expected to be added to the Worker this time. When the processor 82 increases the total amount of virtual resources of the Worker, the total amount of virtual resources of the Worker can be increased according to the upward adjustment increment Δd.

[0173] In some examples of this embodiment, when the processor 82 reduces the total amount of virtual resources of the Worker, if it is determined that the cumulative adjustment amount tc of the virtual resources of the Worker is greater than 0, the corresponding downward adjustment increment of the Worker is determined according to the following formula:

[0174]

[0175] Among them, the value of x is a positive number greater than or equal to 1, which is the fallback multiple. The larger its value is, the larger the absolute value of Δc is;

[0176] Reduce the total amount of virtual resources of the Worker by the downward adjustment increment Δc.

[0177] In some examples of this embodiment, when the processor 82 adjusts the total amount of virtual resources of the Worker according to the resource adjustment request, if it is determined that the resource adjustment request is an upward adjustment request, and the sum of the Worker's cumulative adjustment amount tc of virtual resources and the upward adjustment increment Δd exceeds the adjustment amount upper limit H; or the resource adjustment request is a downward adjustment request, but the Worker's cumulative adjustment amount tc of virtual resources is equal to 0, then the Worker's current total amount of virtual resources is maintained unchanged.

[0178] In some examples of this embodiment, the memory 84 in the computer device 80 can store both the second resource adjustment program and the first resource adjustment program, so the computer device 80 can act as a Master or a Worker.

[0179] In an example of this embodiment, a Spark cluster is also provided. Fig. 9As shown in the schematic diagram, the Spark cluster 9 includes at least one Master 91 and at least one Worker 92, wherein the Master 91 is Figure 8 A computer device having a second resource adjustment program stored in a memory, Worker 92 is Figure 8 A computer device having a first resource adjustment program stored in its memory.

[0180] This embodiment provides a computer program product, including a computer readable device, on which the computer program shown above is stored. In this embodiment, the computer readable device may include the computer readable storage medium shown above.

[0181] It can be seen that those skilled in the art should understand that all or some of the steps, systems, and functional modules / units in the above disclosed methods can be implemented as software (which can be implemented with computer program code executable by a computing device), firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit.

[0182] In addition, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, computer program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. Therefore, the present invention is not limited to any specific hardware and software combination.

[0183] The above contents are further detailed descriptions of the embodiments of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A resource adjustment method, applied to a worker node Worker side in a Spark cluster, the resource adjustment method comprising: Monitor the resource utilization of the physical resources of this Worker in the current monitoring cycle; Evaluate the utilization of the physical resource in the monitoring period according to the resource utilization and the resource utilization critical interval; If the evaluation result indicates that the utilization of the physical resource in the monitoring period is unreasonable, a resource adjustment request is generated; Send the resource adjustment request to the management node Master to request the Master to adjust the total amount of virtual resources of the Worker according to the resource adjustment request; The resource utilization rate of the physical resources of the worker in the current monitoring cycle includes: Collect physical resource statistics of the Worker in the current monitoring period; Determine the instantaneous utilization rate of the physical resource within the monitoring period according to the physical resource statistical data; An average of the instantaneous utilization rates of each monitoring period in the most recent M monitoring periods is determined as the resource utilization rate of the physical resource in the monitoring period, where M is an integer greater than or equal to 1.

2. The resource adjustment method according to claim 1, characterized in that: The evaluating the utilization of the physical resource in the monitoring period according to the resource utilization and the critical interval of the resource utilization includes: Comparing the resource utilization rate of the monitoring period with the upper limit of resource utilization rate and the lower limit of resource utilization rate of the critical interval of resource utilization rate; After determining that the comparison result is any one of the following, it is determined that the utilization of the physical resource in the monitoring period is unreasonable: First, the resource utilization rate of the monitoring period is higher than the resource utilization rate upper limit; Second, the resource utilization rate of the monitoring period is lower than the resource utilization lower limit.

3. The resource adjustment method according to claim 2, characterized in that: After determining that the comparison result is the second type, and before determining that the utilization of the physical resource in the monitoring period is unreasonable, the method further includes: It is determined that the same comparison result appears continuously in the latest N monitoring cycles, where N is an integer greater than or equal to 2.

4. The resource adjustment method according to claim 2, characterized in that: The resource utilization rate includes at least one of the average CPU utilization rate and the average input / output IO utilization rate; If the resource utilization includes an average CPU utilization, the resource utilization critical interval includes an average CPU utilization critical interval, the resource utilization upper limit includes an average CPU utilization upper limit, and the resource utilization lower limit includes an average CPU utilization lower limit; If the resource utilization includes an IO average utilization, the resource utilization critical interval includes an IO average utilization critical interval, the resource utilization upper limit includes an IO average utilization upper limit, and the resource utilization lower limit includes an IO average utilization lower limit.

5. The resource adjustment method according to claim 4, characterized in that: The resource utilization rate includes both the average CPU utilization rate and the average IO utilization rate; after determining that the comparison result is the second type and the same comparison result appears continuously in the latest N monitoring cycles, before determining that the utilization of the physical resource in the monitoring cycle is unreasonable, the following further includes: The utilization of the physical resource in the monitoring period is evaluated in combination with the memory utilization of the Worker.

6. The resource adjustment method according to claim 5, characterized in that: The evaluating the utilization of the physical resource in the monitoring period in combination with the memory utilization of the Worker includes: If it is determined that the idle rate of the Worker memory is not lower than the Free memory safety red line, and it is determined that the available rate of the Worker memory is not lower than the Available memory safety red line, it is determined that the utilization of the physical resources in the monitoring period is unreasonable.

7. The resource adjustment method according to any one of claims 1 to 6, characterized in that: The resource adjustment request can at least reflect the adjustment direction of the requested resource adjustment, and generating the resource adjustment request includes: If the resource utilization is lower than the critical interval of resource utilization, generating an increase request for increasing the total amount of the Worker virtual resources; If the resource utilization is higher than the critical interval of resource utilization, a request for decreasing the total amount of the Worker virtual resources is generated.

8. A resource adjustment method, applied to the Master side in a Spark cluster, the resource management method comprising: Receive a resource adjustment request sent by a Worker, where the resource adjustment request is generated according to a critical interval of resource utilization and a resource utilization of the physical resources of the Worker in a most recent monitoring cycle, indicating that utilization of the physical resources of the Worker in the monitoring cycle is unreasonable; Adjusting the total amount of virtual resources of the Worker according to the resource adjustment request; The resource utilization of the physical resources of the Worker in the most recent monitoring cycle is determined by the Worker through the following steps: Collect physical resource statistics of the Worker in the current monitoring cycle; wherein the current monitoring cycle is the most recent monitoring cycle; Determine the instantaneous utilization rate of the physical resource within the monitoring period according to the physical resource statistical data; An average of the instantaneous utilization rates of each monitoring period in the most recent M monitoring periods is determined as the resource utilization rate of the physical resource in the monitoring period, where M is an integer greater than or equal to 1.

9. The resource adjustment method according to claim 8, characterized in that: The resource adjustment request can at least reflect the adjustment direction of the requested resource adjustment, and adjusting the total amount of virtual resources of the Worker according to the resource adjustment request includes: If the resource adjustment request is an increase request, increase the total amount of virtual resources of the Worker; If the resource adjustment request is a downward adjustment request, the total amount of virtual resources of the Worker is reduced.

10. The resource adjustment method according to claim 9, characterized in that: Before increasing the total amount of virtual resources of the Worker, the method further includes: Determine that the sum of the cumulative adjustment amount tc of the Worker's virtual resources and the upward adjustment increment Δd has not exceeded the upper limit H of the adjustment amount; tc is the difference between the total amount of virtual resources currently available to the Worker and the initial total amount of virtual resources at the start of the game; Δd ​​is the number of virtual resources expected to be added to the Worker this time; Increasing the total amount of virtual resources of the Worker includes: The total amount of virtual resources of the Worker is increased according to the upward adjustment increment Δd.

11. The resource adjustment method according to claim 9, characterized in that: The reducing the total amount of virtual resources of the Worker includes: If the cumulative adjustment amount tc of the virtual resources of the Worker is greater than 0, the corresponding downward adjustment increment of the Worker in this time is determined according to the following formula: tc is the difference between the total amount of virtual resources currently available to the Worker and the initial total amount of virtual resources at the start of the game; the value of x is a positive number greater than or equal to 1, which is the fallback multiple. The larger the value, the larger the absolute value of Δc; The total amount of virtual resources of the Worker is reduced according to the downward adjustment increment Δc.

12. The resource adjustment method according to claim 9, characterized in that: The adjusting the total amount of virtual resources of the Worker according to the resource adjustment request includes: If the resource adjustment request is an upward adjustment request, and the sum of the Worker's cumulative virtual resource adjustment amount tc and the upward adjustment increment Δd exceeds the adjustment amount upper limit H; or the resource adjustment request is a downward adjustment request, but the Worker's cumulative virtual resource adjustment amount tc is equal to 0, then the Worker's current total virtual resource amount is maintained unchanged, and tc is the difference between the Worker's current total virtual resource amount and the initial total virtual resource amount at the start of the game.

13. The resource adjustment method according to any one of claims 8 to 12, characterized in that: The virtual resources include a computing core Core and memory.

14. A computer device, characterized in that: The computer device includes a processor, a memory and a communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the steps of the resource adjustment method as described in any one of claims 1 to 7; or the processor is used to execute one or more computer programs stored in the memory to implement the steps of the resource adjustment method as described in any one of claims 8 to 13.

15. A storage medium, characterized in that: The storage medium stores at least one of a first resource adjustment program and a second resource adjustment program. The first resource adjustment program can be executed by one or more processors to implement the steps of the resource adjustment method as described in any one of claims 1 to 7; the second resource adjustment program can be executed by one or more processors to implement the steps of the resource adjustment method as described in any one of claims 8 to 13.

Images