Distributed resource adjustment method, device and storage medium

Abstract

The present application discloses a distributed resource adjustment method, device and storage medium, which includes: determining the target executor with abnormal memory usage and the memory abnormality type of the target executor based on the received memory usage information of each executor in the server cluster; selecting the target resource adjustment strategy corresponding to the memory abnormality type of the target executor; determining the target memory interval based on the memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory interval; scheduling the task to be executed to the optimized executor for task execution processing. The above scheme can improve the efficiency of resource adjustment.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a distributed resource adjustment method, device and storage medium. Background Art

[0002] Distributed computing engines are widely used in big data computing and analysis. The master node (driver) of the distributed computing engine can divide the computing tasks issued by users into several minimum execution units, and then dispatch these minimum execution units to the slave nodes (executors) for parallel computing.

[0003] However, due to the huge differences in the amount of data processing between different distributed computing tasks in different business scenarios, it is difficult for task developers to accurately assess the exact amount of memory required by the executor of the distributed task. They either set too small a memory size, which causes the executor to have a memory overflow and ultimately leads to task failure, or set too large a memory size, which results in a waste of cluster resources.

[0004] At present, manual experience is often needed to tune the computing memory required for distributed computing tasks based on business scenarios, the amount of data to be processed, and task characteristics. This stage is not only manpower-consuming, but also requires a large amount of computing resources and has low efficiency. Summary of the invention

[0005] The present application at least provides a distributed resource adjustment method, apparatus, device, and computer-readable storage medium.

[0006] The first aspect of the present application provides a distributed resource adjustment method, including: based on the received memory usage information of each executor in the server cluster, determining the target executor with memory usage abnormality in each executor and the memory abnormality type of the target executor; selecting the target resource adjustment strategy corresponding to the memory abnormality type of the target executor; determining the target memory range based on the memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory range; scheduling the task to be executed to the optimized executor for task execution processing.

[0007] In one embodiment, the memory exception type includes a memory overflow, the resource adjustment strategy includes a capacity expansion strategy, the target memory interval is determined based on the memory usage information of the target executor within a preset period, and the optimized executor is created based on the target resource adjustment strategy and the target memory interval, including: in response to the memory exception type being the memory overflow, selecting the capacity expansion strategy as the target resource adjustment strategy; performing memory usage prediction based on the memory usage information of the target executor within a preset period to obtain a target memory interval; creating the optimized executor based on the capacity expansion strategy and the target memory interval, the maximum memory value of the optimized executor being greater than the maximum memory threshold value of the target executor.

[0008] In one embodiment, scheduling the task to be executed to the optimized executor for task execution processing includes: in response to the memory exception type being a memory overflow, scheduling the task to be executed in the target executor to the optimized executor; in response to the failure of scheduling the task to be executed, rolling back the task to be executed.

[0009] In one embodiment, before rolling back the task to be executed in response to the failure of scheduling the task to be executed, the method further includes: obtaining the task scheduling time for scheduling the task to be executed in the target executor to the optimized executor; in response to the task scheduling time being greater than a preset scheduling time threshold, determining that the scheduling of the task to be executed has failed.

[0010] In one embodiment, the memory exception type includes excess memory, the resource adjustment strategy includes a reduction strategy, the target memory range is determined based on the memory usage information of the target executor within a preset period, and the optimized executor is created based on the target resource adjustment strategy and the target memory range, including: in response to the memory exception type being excess memory, selecting the reduction strategy as the target resource adjustment strategy; performing memory usage prediction based on the memory usage information of the target executor within a preset period to obtain a target memory range; creating the optimized executor based on the reduction strategy and the target memory range, the maximum memory value of the optimized executor being less than or equal to the minimum memory threshold value of the target executor.

[0011] In one embodiment, before creating the optimized executor based on the scaling-in strategy and the target memory range, the method further includes: in response to the memory exception type being the excess memory, stopping allocating tasks to the target executor; in response to completion of execution of the allocated tasks in the target executor, shutting down the target executor.

[0012] In one embodiment, scheduling the tasks to be executed to the optimizing executor for task execution processing includes: stopping allocating tasks to the target executor; in response to completion of execution of the assigned tasks in the target executor, shutting down the target executor; and allocating the tasks to be executed to the optimizing executor.

[0013] In one embodiment, the memory usage prediction is performed based on the memory usage information of the target executor within a preset period to obtain a target memory interval, including: determining the right boundary of the target memory interval based on the maximum memory demand of the target executor within the preset period; and determining the left boundary of the target memory interval based on the minimum memory demand of the target executor within the preset period.

[0014] The second aspect of the present application provides a distributed resource adjustment device, including: an exception determination module, which is used to determine the target executor with memory usage exception in each executor and the memory exception type of the target executor based on the memory usage information of each executor in the server cluster; a strategy selection module, which is used to select the target resource adjustment strategy corresponding to the memory exception type of the target executor; a memory optimization module, which is used to determine the target memory range based on the memory usage information of the target executor within a preset period, and create an optimized executor based on the target resource adjustment strategy and the target memory range; a task scheduling module, which is used to schedule the task to be executed to the optimized executor for task execution processing.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned distributed resource adjustment method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above-mentioned distributed resource adjustment method when the program instructions are executed by a processor.

[0017] The above scheme receives the memory usage information of each executor in the server cluster through the driver in the server cluster to determine the target executor with abnormal memory usage in each executor and the memory abnormality type of the target executor; selects the corresponding target resource adjustment strategy according to the memory abnormality type of the target executor; analyzes the memory usage information of the target executor within a preset period to determine the target memory interval required for the normal operation of the task to be executed in the target executor, and creates an optimized executor based on the target resource adjustment strategy and the target memory interval; and schedules the task to be executed to the optimized executor for task execution processing. In this way, the number of executors and the resources used by the executors can be adjusted, so as to realize the multi-dimensional elastic scaling of the executor computing resources, improve the efficiency of computing resource allocation, and improve the overall resource utilization rate of the distributed cluster, reducing the problem of resource fragmentation.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 is a flowchart of an exemplary embodiment of a distributed resource adjustment method of the present application;

[0021] Figure 2 It is a schematic diagram of expanding the capacity of an executor with memory overflow in the distributed resource adjustment method of the present application;

[0022] Figure 3 It is a schematic diagram of scaling down an executor with excess memory in the distributed resource adjustment method of the present application;

[0023] Figure 4 is a block diagram of a distributed resource adjustment device shown in an exemplary embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0025] Figure 6 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0026] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0027] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0028] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent the following three pieces of information: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated with each other are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of types. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0029] For ease of understanding, an exemplary description of one of the applicable scenarios of this application is now given. Distributed computing engines are widely used in the fields of big data computing and analysis. The master node of the distributed computing engine (also referred to as the driver node in this article) can divide the computing tasks posted by the user into several minimum execution units, and each execution unit can be regarded as a computing task (also referred to as a Task), and then dispatch these minimum execution units to the slave nodes (also referred to as executor nodes) for parallel computing. This is a conventional task execution process in the field of big data technology, which will not be elaborated here.

[0030] For example, the current mainstream distributed computing engines, such as Apache Spark, Apache Flink, ApacheStorm, etc., all adopt a similar master-slave structure. The master node (driver node) is responsible for computing task decomposition, scheduling, and computing resource application; the slave node (executor node) is responsible for actual data calculation. For example, in Spark, the master node is Driver and the slave node is Executor; in Flink, the master node is JobManager and the slave node is TaskManager; in Storm, the master node is Supervisor and the slave node is Worker. For the sake of simplicity, the master node of the master-slave structure distributed computing engine will be uniformly referred to as Driver (driver) and the slave node as Executor (executor) in the following text. Generally speaking, the distributed computing engine will divide the user-submitted job into the smallest execution unit at the master node, and these smallest execution units will be called Task later. Task is scheduled to different slave nodes for execution by the Task scheduler of the master node, and has a fault-tolerant mechanism. When the Task fails to execute, the master node can reschedule the Task to other slave nodes for execution. At the same time, when an abnormality occurs in a slave node, the master node can also reapply for a new slave node from the cluster resource manager to ensure that computing resources meet user needs.

[0031] However, due to the huge difference in the amount of data required to be processed by distributed computing tasks in different business scenarios, it is difficult for task developers to accurately evaluate the exact amount of memory required by the distributed task executor. They usually roughly estimate the memory resources required for task execution and set a memory threshold for each node. In fact, this threshold setting method either sets too small a memory, causing the executor to have a memory overflow problem, which ultimately leads to task failure; or sets too large a memory, resulting in excess resources and resource waste. For example, if there is a serious data volume skew for a period of time, or a surge in data volume on a certain day, the memory required for a task in the distributed task to calculate exceeds the Executor memory limit set by the user, which will cause the Executor memory overflow. And even if there are fault-tolerant mechanisms such as Executor reapplication and Task retry, since the memory required for Task execution has exceeded the Executor memory limit, even if multiple retries are made, it will not succeed. Eventually, the task will fail because the number of Executor failures reaches the threshold or the number of Task retries reaches the threshold.

[0032] The solution often requires manual experience to continuously optimize the memory resources required for distributed computing tasks based on business scenarios, data volume, and task characteristics. This stage is not only very labor-intensive, but also requires a large amount of computing resources to be invested in experiments, and is inefficient.

[0033] See also Figure 1 , Figure 1 1 is a flow chart of an exemplary embodiment of a distributed resource adjustment method of the present application. The method of the present application is applied to a driver of a server cluster. Specifically, the following steps may be included:

[0034] Step S110, based on the received memory usage information of each executor in the server cluster, determining a target executor with memory usage abnormality among the executors and a memory abnormality type of the target executor.

[0035] Memory usage information mainly represents the memory usage requirements of the executor during the task execution process, and may include data used to measure memory load such as the memory usage value or memory usage rate of the executor during the task execution process, which may be determined based on the executor during the task execution process or based on the executor within a certain period. For example, if the memory upper limit applied by an executor A is 2GB, and the executor A uses a total of 1GB of memory resources when executing the first task, then the memory usage value of the executor A when executing the first task is 1GB, and the memory usage rate is 50%. In addition, memory usage information may also include the memory requirements when the task to be executed is executed. For example, in the above example, the memory requirement for executing the first task is 1GB; in addition, if the memory requirement of the second task is 3GB, which exceeds the memory upper limit of executor A, then executor A cannot execute the second task and cannot obtain the memory usage value or memory usage rate when executing the second task, but can judge the memory overflow based on the memory requirement of the second task and the memory upper limit of executor A, then executor A will report the memory exception type of memory overflow.

[0036] It should be noted that in this application, the memory exception type may include not only the problem of memory overflow, but also the problem of excess memory. Excess memory mainly refers to the fact that during the execution of a task by the executor, the memory load of the executor is less than the preset first load threshold for a long time. In this case, it can be determined that some memory resources in the executor are not used, that is, there is a problem of resource waste. Still using the above example for illustration, if the memory resource of executor A is 2GB, but the memory usage value of executor A each time it executes a task is only 1GB, that is, the memory usage rate of executor A each time it executes a task is 50%, if the first load threshold set for executor A is 70%, it can be determined that executor A has an excess memory problem. It should be noted that different executors may be provided with different load thresholds, which are not limited here. The first load threshold is a threshold used to measure whether the executor has an excess memory problem. And to measure whether the executor has a memory overflow problem, it can be used as the upper limit of the memory of the executor as the second load threshold, or to set a second load threshold separately, which will not be elaborated here. Among them, the second load threshold should be greater than the first load threshold.

[0037] It should also be noted that the memory usage information of the executor may be obtained by the driver actively requesting the executor, and / or the executor actively reporting to the driver, which is not limited here.

[0038] In summary, the memory usage information of each executor can be used to analyze whether there is an executor with abnormal memory usage among the executors, and the executor with abnormal memory usage is determined as the target executor and the memory exception type of the target executor is obtained.

[0039] Step S120: Select a target resource adjustment strategy corresponding to the memory exception type of the target executor.

[0040] In conjunction with the above steps, in the resource adjustment method of the present application, different resource adjustment strategies need to be selected for target executors with different memory exception types. Since there can be multiple executor nodes in a server cluster, the memory exception types of different target executors can be the same or different, which is not limited here.

[0041] Exemplarily, the resource adjustment strategy of the present application includes an expansion strategy and a contraction strategy. If the memory exception type of the target executor is memory overflow, the expansion strategy is selected as the target resource adjustment strategy, and one or more optimized executors with larger memory are created to execute tasks to achieve resource expansion. If the memory exception type of the target executor is excess memory, the contraction strategy is selected as the target resource adjustment strategy, and one or more optimized executors with smaller memory are created to execute tasks to achieve resource contraction.

[0042] Step S130, determining a target memory range based on memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory range.

[0043] In conjunction with the above steps, the resource adjustment strategy in the resource adjustment method of the present application is to replace the target executor with abnormal memory usage by creating one or more optimized executors to achieve resource adjustment. It should be noted that the present application implements elastic scaling of the memory resources of the executor by setting a memory interval in the optimized executor. Among them, the memory interval of the optimized executor can also be understood as a memory load interval.

[0044] The preset cycle refers to a pre-specified periodic time period, for example, determining the target memory range based on the memory usage information of the target executor within each hour, or determining the target memory range based on the memory usage information of the target executor within one or more time periods, or determining the target memory range based on the memory usage information of the target executor during the entire period of one or more task execution processes, or determining the target memory range based on the memory usage information of the target executor during a partial period of one or more task executions, which is not limited here.

[0045] Exemplarily, the upper memory limit (or memory resource amount) of the optimized executor created according to the expansion strategy should be greater than the upper memory limit (memory resource amount) of the corresponding target executor. The upper memory limit of the optimized executor created according to the reduction strategy should be less than the upper memory limit of the corresponding target executor. In addition, in the process of creating the optimized executor, the present application will also set the lower memory limit of the optimized executor, and the memory load range of the optimized executor is formed by the upper memory limit and the lower memory limit.

[0046] Step S140, dispatching the task to be executed to the optimization executor for task execution processing.

[0047] In conjunction with the above steps, after the optimization executor is successfully created, the tasks to be executed in the current server cluster can be scheduled to the optimization executor for task execution processing.

[0048] It should be noted that the pending tasks that are not assigned to the target executor may be scheduled to the optimized executor for execution, or the pending tasks that are assigned to the target executor may be scheduled to the optimized executor for execution, which is not limited here.

[0049] It can be seen that the present application receives the memory usage information of each executor in the server cluster through the driver in the server cluster to determine the target executor with abnormal memory usage in each executor and the memory abnormality type of the target executor; selects the corresponding target resource adjustment strategy according to the memory abnormality type of the target executor; analyzes the memory usage information of the target executor within a preset period, determines the target memory interval required for the normal operation of the task to be executed in the target executor, and creates an optimized executor based on the target resource adjustment strategy and the target memory interval; schedules the task to be executed in the target executor to the optimized executor for task execution processing. In this way, the number of executors and the resources used by the executors can be adjusted, and the multi-dimensional elastic scaling of the executor computing resources can be realized, the efficiency of computing resource allocation can be improved, and the overall resource utilization rate of the distributed cluster can be improved, and the resource fragmentation problem can be reduced.

[0050] Based on the above embodiments, the embodiment of the present application describes the steps of determining a target memory interval based on the memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory interval. Among them, the memory exception type includes memory overflow, and the resource adjustment strategy includes a capacity expansion strategy. Specifically, the method of this embodiment includes the following steps:

[0051] In response to the memory exception type being memory overflow, a capacity expansion strategy is selected as the target resource adjustment strategy; memory usage prediction is performed based on the memory usage information of the target executor within a preset period to obtain a target memory range; an optimized executor is created based on the capacity expansion strategy and the target memory range, and the maximum memory value of the optimized executor is greater than the maximum memory threshold of the target executor.

[0052] In combination with the above-mentioned embodiments, this embodiment mainly explains the solution to the memory overflow problem of the target executor.

[0053] Specifically, after the driver and executor in the server cluster are successfully started, the driver will schedule tasks for each executor to execute. When the executor is executing the task it is responsible for, if a memory overflow occurs, the memory exception information (memory usage information) will be reported. If the driver determines that the target executor has a memory overflow (which may include heap memory overflow and / or off-heap memory overflow) based on the memory usage information reported by each executor, it will apply for memory resources from the cluster resource manager (such as Yarn, K8S, etc.) in the current server cluster and create an optimized executor. Among them, the memory resources applied for will be larger than the memory resources of the target executor.

[0054] Exemplarily, the process of applying for memory resources may include, but is not limited to, using pre-set rules, algorithms, neural network models and other methods (such as various existing traditional regression algorithms, machine learning algorithms, etc., which are not limited here), to predict memory usage based on the memory usage information of the target executor in the previous preset period, and obtain a target memory range, which is equivalent to inferring the reasonable in-heap / off-heap memory while ensuring the normal operation of subsequent tasks based on the previous memory usage of the target executor. And use the predicted target memory range to create an optimized executor. Among them, for the target executor with a memory overflow problem, it is necessary to create an optimized executor with a larger memory than the target executor based on the expansion strategy. Therefore, the maximum memory value of the optimized executor is greater than the maximum memory threshold of the target executor, so that the task that causes memory overflow in the target executor can be executed normally in the optimized executor.

[0055] Based on the above embodiments, the embodiment of the present application describes the steps of scheduling the tasks to be executed to the optimized executor for task execution processing. Specifically, the method of this embodiment includes the following steps:

[0056] In response to the memory exception type being memory overflow, the task to be executed in the target executor is scheduled to the optimized executor; in response to the scheduling failure of the task to be executed, the task to be executed is rolled back.

[0057] In conjunction with the foregoing embodiment, if it is detected that the target executor has a memory overflow problem, after the optimized executor is created, the pending tasks in the target executor can be scheduled to be executed in the optimized executor. Among them, the pending tasks in the target executor can be tasks that the target executor failed to execute due to memory overflow, and / or tasks that the target executor can execute normally, which is not limited here. In addition, in addition to scheduling the pending tasks in the target executor to the optimized executor, it is also possible to pause or stop allocating tasks to the target executor, and to allocate pending tasks that are not allocated to the target executor to the optimized executor, etc., which will not be elaborated here.

[0058] In response to the failure of scheduling the pending task (which may be due to the failure of the optimized executor to start, such as insufficient server cluster resources, high queue load, slow container startup, etc.), the pending task needs to be rolled back so that it can be scheduled to other executor nodes in the server cluster (preferentially other executors in the server cluster except the target executor) to ensure that the task is not blocked. Alternatively, in response to the failure of the pending task after being scheduled to the optimized executor, the failed task can also be rolled back.

[0059] For example, reference may also be made to Figure 2 As shown, Figure 2 This is a schematic diagram of the expansion and adjustment of the executor with memory overflow in the distributed resource adjustment method of the present application. In the entire application scenario, if the driver node (driver) detects that the executor node (target executor) has a memory overflow exception, it will adjust the size of the in-heap / off-heap memory when applying to create a new executor node (optimized executor), and apply for an executor node (optimized executor) with larger memory than the original executor (target executor). And for tasks that failed to execute on the original executor node (tasks to be executed), they can be preferentially scheduled to the executor node after the memory expansion for execution. If the optimized executor fails to start successfully for a long time, the task can be transferred back to the original executor node for task execution (preferentially scheduled to other original executor nodes in the server cluster except the target executor), so as to maximize the probability of successful task restart.

[0060] The main process steps in the above example scenario may include but are not limited to:

[0061] 1. After the driver and executor in the server cluster are started, the driver assigns scheduling tasks to each executor for execution. Here, the original executor nodes in the server (executor 1 of xGB and executor 2 of xGB) are shown as examples.

[0062] 2. If a memory overflow exception occurs when executor 1 node is executing tasks 1 and 2, the memory exception information will be reported to the cluster resource manager.

[0063] 3. The driver's executor scheduler periodically obtains the status of each executor node from the cluster resource manager and detects the memory overflow exception type that occurs in executor 1.

[0064] 4. The driver's task scheduler obtains the reason for the task failure from the executor scheduler and determines that executor 1 has a memory overflow.

[0065] 5. The driver's executor scheduler uses a series of rules and algorithms to predict reasonable on-heap / off-heap memory based on the original memory, operation indicators and other memory usage information of executor 1 where memory overflow occurs, and uses the predicted on-heap / off-heap memory when applying for a new executor 3 (optimized executor). Among them, if a cluster resource manager that can configure the RequestMemory and LimitMemory memory of the executor container is used in a specific application scenario. For example, Kubernetes, which often uses two types of restrictions, Request and Limit, to allocate resources, can further control the memory interval setting of executor 3. Among them, RequestMemory represents the minimum resource requirement used by the executor container, which is used as the judgment dependency of resource allocation during container scheduling, which is equivalent to the minimum memory value of the optimized executor; and LimitMemory represents the maximum value of resources that the executor container can use. Setting it to 0 means that there is no upper limit on the resources that can be used, which is equivalent to the maximum value of memory that the optimized executor has. Request can ensure that the executor container has enough resources to run, while Limit prevents an executor container from using resources without limit, causing other executor containers to crash. For example, the RequestMemory of executor 3 can be configured as the memory of executor 1, and the LimitMemory of executor 3 can be configured as the predicted memory, thereby reducing the memory waste caused by increasing the memory of the executor. It should be noted that this application uses the method of setting the target memory range for the optimized executor, so that the resource elastic scaling adjustment strategy is not limited to the k8s cluster management system, thereby improving versatility.

[0066] 6. The cluster resource manager creates zGB's executor 3 node (optimized executor).

[0067] 7. When the task scheduler of the driver schedules a task that fails to execute due to memory overflow, it first schedules such task to the executor 3 with expanded memory for execution.

[0068] 8. If executor 3 node cannot be started for a long time, the task will be rolled back to the original executor node other than executor 1 in the server cluster (for example, executor 2) to ensure that the task is not blocked. Among them, it can be determined whether the memory of each original executor is greater than the memory of the executor that failed to execute the task (for example, the memory of executor 2 is greater than that of executor 1), and the preferred executor is obtained, and then the task is rolled back to the preferred executor.

[0069] It should be noted that if there are already assigned and waiting-to-be-executed tasks 3 and 4 in executor 2, when tasks 1 and 2 are rolled back to executor 2, tasks 3 and 4 can be executed first, and then tasks 1 and 2; or the task execution order can be rearranged according to the task priorities of tasks 1-4, which is not limited here.

[0070] Based on the above embodiments, the embodiment of the present application describes the steps before the task to be executed is rolled back in response to the failure of scheduling the task to be executed. Specifically, the method of this embodiment includes the following steps:

[0071] Obtaining a task scheduling time for scheduling the task to be executed in the target executor to the optimized executor; in response to the task scheduling time being greater than a preset scheduling time threshold, determining that the scheduling of the task to be executed fails.

[0072] As described in conjunction with the above-mentioned embodiment, when scheduling the to-be-executed tasks in the target executor to the optimized executor, there may also be a problem of scheduling failure or a problem of execution failure after being scheduled to the optimized executor.

[0073] Therefore, during the scheduling process, the task scheduling time or the startup time of the optimized executor can be recorded and compared with the preset scheduling time threshold, thereby indicating whether the optimized executor is successfully started within a reasonable time and receives the task to be executed scheduled from the target executor. In addition, the task execution time of the optimized executor can also be recorded, for example, the time from the time when the optimized executor receives the task to be executed to the time when the optimized executor starts to execute the task, thereby judging whether the optimized executor can execute the task normally. The time judgment methods of the above two processes can be applied either one or both at the same time, which is not limited here.

[0074] Based on the above embodiment, the embodiment of the present application further describes the steps of determining the target memory range based on the memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory range. Among them, the memory exception type includes excess memory, and the resource adjustment strategy includes a shrinking strategy. Specifically, the method of this embodiment includes the following steps:

[0075] In response to the memory exception type being excess memory, a scaling-down strategy is selected as the target resource adjustment strategy; memory usage prediction is performed based on the memory usage information of the target executor within a preset period to obtain a target memory range; an optimized executor is created based on the scaling-down strategy and the target memory range, and the maximum memory value of the optimized executor is less than or equal to the minimum memory threshold value of the target executor.

[0076] In combination with the above-mentioned embodiments, this embodiment mainly describes the situation where the memory exception type is excess memory. If the target executor has an excess memory problem, it means that most of the memory resources of the target executor are idle, resulting in memory waste. Therefore, in the resource adjustment method of this application, a shrinking strategy is selected to shrink the target executor.

[0077] For details, please refer to the above explanation of the expansion strategy. In the process of implementing the reduction strategy, the same method (preset rules, algorithms, etc.) can be used to predict the memory usage information of the target executor within the preset period to obtain the target memory range. Then, based on the reduction strategy and the target memory range, a new optimized executor with smaller memory than the target executor is created.

[0078] Based on the above embodiments, the present embodiment of the application describes the steps before creating an optimized executor based on the scaling-in strategy and the target memory range. Specifically, the method of this embodiment includes the following steps:

[0079] In response to the memory exception type being excess memory, the task allocation to the target executor is stopped; in response to the completion of the execution of the allocated tasks in the target executor, the target executor is closed.

[0080] In conjunction with the above-mentioned embodiment, when it is determined that the target executor's memory exception type is excess memory, the driver needs to stop allocating tasks to the target executor. At this time, it is necessary to ensure that the target executor completes the execution of the assigned tasks in its own node. When the assigned tasks in the target executor are completed, the driver can close the target executor, thereby releasing the resources occupied by the target executor. In the subsequent implementation process, the driver can allocate the tasks to be assigned to the optimized executor.

[0081] Among them, the judgment conditions for memory surplus may include but are not limited to that the memory usage values ​​of the executor within a preset time period are all less than a preset memory surplus threshold, or the statistical value of the memory usage value of the executor within the preset time period (such as one or more of the mean, median, mode, etc.) is less than the preset memory surplus threshold, then it is determined that the executor has a memory surplus problem.

[0082] For example, reference may be made to Figure 3 As shown, Figure 3This is a schematic diagram of scaling down an executor with excess memory in the distributed resource adjustment method of the present application. In the entire application scenario, if the driver node detects that an executor node (target executor) has an excess memory exception, it will stop allocating tasks to the target executor until the target executor completes the execution of the assigned tasks, and then the driver shuts down and stops the target executor. Memory usage is predicted based on the previous memory usage information of the target executor to obtain the target memory range, and when applying to create a new executor node (optimized executor), the size of the in-heap / off-heap memory is adjusted based on the target memory range, and an application is made to create an executor node (optimized executor) with smaller memory than the original executor (target executor). After the creation is complete, the optimized executor can be started and the driver can assign appropriate tasks to be executed to the optimized executor.

[0083] The main process steps in the above example scenario may include but are not limited to:

[0084] 1. After the driver and each executor in the server cluster are started, the driver assigns scheduling tasks to the executor nodes with original memory resources (such as executor 4 of xGB and executor 5 of xGB) for execution.

[0085] 2. During the execution of tasks, these executors will periodically report the on-heap / off-heap memory usage to the driver.

[0086] 3. If the driver detects that a certain executor (for example, executor 4) is in a state of low memory load pressure for a long time (determining that there is an excess memory abnormality), the task scheduler is notified to stop scheduling tasks to the executor 4 node.

[0087] 4. After waiting for all tasks on executor 4 to be executed, the driver sends a request to executor 4 to shut down and stop executor 4.

[0088] 5. The driver uses a series of rules and algorithms to predict reasonable on-heap / off-heap memory based on the original memory, operating indicators and other memory usage information of executor 4, and uses the predicted reduced heap / off-heap memory when applying for a new executor 6 (optimized executor). If a cluster resource manager that can configure the RequestMemory and LimitMemory memory of the executor container is used in a specific application scenario. For example, Kubernetes, the memory range setting of executor 6 can be further controlled. For example, the RequestMemory of executor 6 can be configured as the predicted memory value, and the LimitMemory can be configured as the original memory of executor 4, so that the memory of the optimized executor can be elastically scaled in the target memory range of [RequestMemory, LimitMemory], and the risk of memory overflow exceptions after reducing the executor memory is reduced.

[0089] 6. The cluster resource manager creates the optimized executor 6 of yGB.

[0090] 7. The task scheduler in the driver schedules subsequent suitable (memory matching) tasks to the optimized executor.

[0091] Based on the above embodiments, the embodiment of the present application describes the steps of scheduling the tasks to be executed to the optimized executor for task execution processing. Specifically, the method of this embodiment includes the following steps:

[0092] Stop allocating tasks to the target executor; in response to the completion of the execution of the assigned tasks in the target executor, close the target executor; and allocate the tasks to be executed to the optimized executor.

[0093] In combination with the above-mentioned embodiment, this embodiment provides another implementable method different from the above-mentioned embodiment. In the above-mentioned embodiment, the driver first stops allocating tasks to the target executor until the target executor completes the execution of the assigned tasks, and then the driver closes the target executor and creates an optimized executor for receiving and executing the tasks subsequently allocated and scheduled by the driver.

[0094] In this embodiment, the driver first creates an optimized executor, and then stops assigning tasks to the target executor until the target executor completes the execution of the assigned tasks. The driver then closes the target executor and assigns subsequent tasks to be executed to the optimized executor.

[0095] The specific implementation process can be selected according to the overall memory information of the server cluster. For example, if the current remaining memory information of the server cluster is not enough to apply for the creation of an optimized driver, the optimized executor can be created after the target executor completes the assigned tasks and shuts down. If the current remaining memory information of the server cluster is sufficient to apply for the creation of an optimized driver, the optimized executor can be created first, and then the assigned tasks to the target executor can be stopped until the target executor completes the assigned tasks, and the driver shuts down the target executor. Or "create an optimized executor" and "stop assigning tasks to the target executor until the target executor completes the assigned tasks, and the driver shuts down the target executor" can be performed simultaneously. It can be understood that if the current remaining memory information of the server cluster is sufficient to apply for the creation of an optimized driver, it is also possible to wait for the target executor to complete the assigned tasks and shut down before creating the optimized executor.

[0096] It should be noted that after the optimization executor is created, if the target executor has not completed its assigned tasks, the driver can assign other suitable tasks to be executed to the optimization executor while waiting for the target executor to complete the assigned tasks.

[0097] In addition, it can also be realized that if the optimization executor is created and the target executor has not completed its assigned task, the driver can record the current task progress of the task, schedule the task in the target executor to the optimization executor and continue to execute it based on the current task progress. At this time, the target executor can be directly shut down and stopped to speed up resource recovery efficiency. Among them, if the task can be executed in stages, it can be determined based on the current task progress that the tasks that have not been executed in the task are transferred to the optimization executor for processing. If the task cannot be executed in stages, it is necessary to transfer the entire task to the optimization executor for reprocessing, or still let the target executor complete the task. It can also be determined based on the current task progress and task progress threshold of the task. For example, if the current task progress is 10% less than the preset task progress threshold of 30%, the entire task is transferred to the optimization executor for reprocessing. If the current task progress is 90% greater than the task progress threshold, the target executor is still allowed to complete the task.

[0098] It can be seen from this that when server cluster resources are scarce, creating an optimized executor with more reasonable memory to execute tasks in the target executor with excess memory problems can release the wasted memory resources of the target executor in time and avoid the allocation and execution of other tasks in the cluster being affected by memory fragmentation problems.

[0099] Based on the above embodiment, the embodiment of the present application describes the steps of predicting memory usage based on the memory usage information of the target executor within a preset period to obtain the target memory range. Specifically, the method of this embodiment includes the following steps:

[0100] The right boundary of the target memory interval is determined based on the maximum memory demand of the target executor within the preset period; and the left boundary of the target memory interval is determined based on the minimum memory demand of the target executor within the preset period.

[0101] The maximum memory requirement and the minimum memory requirement refer to the memory values ​​that the executor needs to use when executing a task. Therefore, the maximum memory requirement and the maximum memory value of the executor can be the same or different, and the minimum memory requirement and the minimum memory value of the executor can be the same or different. For example, the maximum memory value of the executor is 2GB, but when executing a task, its maximum memory requirement is 3GB, so a memory overflow problem will occur.

[0102] In conjunction with the foregoing embodiments, the resource adjustment method of the present application includes an expansion strategy and a contraction strategy, and both strategies involve a memory prediction method. Among them, the memory prediction method of the present application can refer to the memory prediction value obtained by the existing regression algorithm or machine learning algorithm, or can be predicted based on preset custom rules, which is not limited here. The memory prediction methods used in the expansion strategy and the contraction strategy can be the same or different, which is not limited in the present application.

[0103] On the one hand, there is an implementation method including: determining the right boundary of the target memory interval based on the maximum memory requirement of the target executor within a preset period; determining the left boundary of the target memory interval based on the minimum memory requirement of the target executor within the preset period. For example, during one or more task executions of the target executor, the minimum memory required by the target executor is 1GB, and the maximum memory required is 3GB, then the target memory interval of the optimized executor can be determined as [1,3] or a range slightly larger than [1,3], which is not limited here.

[0104] On the other hand, there is another possible implementation method including: for a target executor with memory overflow, the memory upper limit of the target executor is small, so the memory upper limit of the target executor can be set as the memory lower limit of the optimization executor (that is, the left boundary of the target memory interval); the right boundary of the target memory interval is determined by the maximum value of the memory demand of the target executor within the preset period. For example, if the memory upper limit of the target executor is 2GB, and its maximum memory demand within the preset period is 3GB, the target memory interval of the optimization executor can be [2, 3]. Similarly, for a target executor with excess memory, the memory upper limit of the target executor is large, so the memory upper limit of the target executor can be set as the memory upper limit of the optimization executor (that is, the right boundary of the target memory interval), or a value less than the memory upper limit of the target executor is set as the memory upper limit of the optimization executor; and the left boundary of the target memory interval is determined by the minimum value of the memory demand of the target executor within the preset period. For example, if the upper memory limit of the target executor is 2GB, and its minimum memory requirement within the preset period is 1GB, then the target memory range of the optimized executor can be [1, 2] or [1, 2) or [1, 1.8], etc., which can be set according to the actual task execution situation.

[0105] On the other hand, there is another possible implementation method including: pre-configuring an algorithm model for computing resource adjustment strategy, which can be one or more computing judgment rules, or a memory prediction value obtained by prediction based on various traditional regression algorithms and machine learning algorithms, and its specific form is not limited. In general, the input parameters of the strategy or algorithm are computing resource parameter type, computing resource parameter adjustment direction (capacity expansion or contraction), operating indicators when executing tasks, etc., and the output parameters are reasonably predicted computing resource parameters (memory prediction values). Exemplarily, when a target executor with memory overflow is detected, the memory usage of the target executor in the previous period of time is analyzed. If the memory usage is on an upward trend, the memory demand for subsequent tasks to be executed will become larger and larger, so more predictions are needed for optimizing the memory upper limit of the executor. Similarly, when a target executor with excess memory is detected, the memory usage of the target executor in the previous period of time is analyzed. If the memory usage is on a downward trend, the memory demand for subsequent tasks to be executed will become smaller and smaller, so less predictions are needed for optimizing the memory upper limit of the executor. When a target executor with excess memory is detected, the memory usage of the target executor over a period of time is analyzed. If the memory usage shows a stable trend (which can be determined based on the slope of data changes, the amplitude of data change differences, etc., and is not limited here), it means that the memory demand of subsequent tasks to be executed is stable near this situation. Therefore, the memory upper limit of the optimized executor can be determined based on the memory usage.

[0106] It should also be noted that, in the process of memory prediction, the right boundary of the target memory range can be obtained by averaging multiple maximum memory demand values ​​reached during the execution of multiple tasks or at multiple time points (for example, average value calculation or weighted average calculation, etc. are not limited). Similarly, the left boundary of the target memory range can be obtained by averaging multiple minimum memory demand values ​​reached during the execution of multiple tasks or at multiple time points.

[0107] In summary, through the resource adjustment method of the present application, the user does not need to set specific in-heap / off-heap memory, number of executors and other configuration information of the fixed executor node in the server cluster, but relies on the system to automatically generate (or customize the preset) the target memory range of the executor node (which may include in-heap memory range (executor.min.memory, executor.max.memory), off-heap memory range (executor.min.memoryOverhead, executor.max.memoryOverhead)); number of executor nodes range (executor.min.instances, executor.max.instances) and other parameters, and can also specify adjustment strategies for various computing resource parameters (number of cores, memory, number of executor instances), and can also select the strategy provided by the system by default.

[0108] Based on the above embodiment, the user can configure the target memory range of a single executor without specifying a specific memory value. During the specific implementation process, the memory of the executor will be automatically adjusted within the target memory range according to the task execution status, which greatly reduces the difficulty of user parameter setting, reduces the waste of cluster resources, and improves the stability of the task.

[0109] This application combines the original resource adjustment capabilities of the distributed computing engine to form a multi-dimensional elastic scaling solution for distributed computing engines. When memory overflow occurs, the executor nodes in the server cluster can be increased and the memory resource space in the executor nodes can be expanded; when memory surplus occurs, the memory resource space in the executor nodes can be reduced. Distributed cluster engines with different master and slave structures can achieve multi-dimensional elastic scaling of computing resources based on the ideas of this proposal.

[0110] Based on the multi-dimensional elastic scaling solution of this application, the number of executors can be increased or decreased, as well as the expansion and contraction of executor resources, so that the resource adjustment and allocation process can be carried out at a finer granularity, thereby achieving the effect of reducing the resource fragmentation rate of the cluster. The system's automatic planning and generation process reduces the difficulty of users configuring computing resource parameters, improves the stability of task operation, reduces task operation and maintenance costs, and achieves the result of reducing costs and increasing efficiency.

[0111] It should be further explained that the execution subject of the distributed resource adjustment method may be a distributed resource adjustment device, for example, the distributed resource adjustment method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the distributed resource adjustment method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0112] Figure 4 FIG. 1 is a block diagram of a distributed resource adjustment device according to an exemplary embodiment of the present application. Figure 4 As shown, the exemplary distributed resource adjustment device 400 includes: an abnormality determination module 410, a strategy selection module 420, a memory optimization module 430 and a task scheduling module 440. Specifically:

[0113] The abnormality determination module 410 is used to determine a target executor with abnormal memory usage among the executors and a memory abnormality type of the target executor based on the received memory usage information of each executor in the server cluster.

[0114] The strategy selection module 420 is used to select a target resource adjustment strategy corresponding to the memory exception type of the target executor.

[0115] The memory optimization module 430 is used to determine a target memory range based on memory usage information of a target executor within a preset period, and to create an optimized executor based on a target resource adjustment strategy and the target memory range.

[0116] The task scheduling module 440 is used to schedule the tasks to be executed to the optimization executor for task execution processing.

[0117] In this exemplary distributed resource adjustment device, the driver in the server cluster receives the memory usage information of each executor in the server cluster to determine the target executor with abnormal memory usage in each executor and the memory abnormality type of the target executor; select the corresponding target resource adjustment strategy according to the memory abnormality type of the target executor; analyze the memory usage information of the target executor within a preset period to determine the target memory interval required for the normal operation of the task to be executed in the target executor, and create an optimized executor based on the target resource adjustment strategy and the target memory interval; schedule the task to be executed to the optimized executor for task execution processing. In this way, the number of executors and the resources used by the executors can be adjusted, and the multi-dimensional elastic scaling of the executor computing resources can be achieved, the efficiency of computing resource allocation can be improved, and the overall resource utilization rate of the distributed cluster can be improved, and the resource fragmentation problem can be reduced.

[0118] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0119] The functions of each module can be found in the distributed resource adjustment method implementation example, which will not be described in detail here.

[0120] See also Figure 5 , Figure 5 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above-mentioned distributed resource adjustment method embodiments. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0121] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned distributed resource adjustment method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0122] In this exemplary electronic device, the memory usage information of each executor in the server cluster is received by the driver in the server cluster to determine the target executor with abnormal memory usage in each executor and the memory abnormality type of the target executor; the corresponding target resource adjustment strategy is selected according to the memory abnormality type of the target executor; the target memory interval required for the normal operation of the task to be executed in the target executor is determined by analyzing the memory usage information of the target executor within a preset period, and an optimized executor is created based on the target resource adjustment strategy and the target memory interval; the task to be executed is scheduled to the optimized executor for task execution processing. In this way, the number of executors and the resources used by the executors can be adjusted, and the multi-dimensional elastic scaling of the executor computing resources can be realized, the computing resource allocation efficiency can be improved, and the overall resource utilization rate of the distributed cluster can be improved, and the resource fragmentation problem can be reduced.

[0123] See also Figure 6 , Figure 6 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above-mentioned distributed resource adjustment method embodiments.

[0124] In this exemplary storage medium, by running the program instructions in the storage medium, the driver in the server cluster receives the memory usage information of each executor in the server cluster, and determines the target executor with abnormal memory usage in each executor and the memory abnormality type of the target executor; selects the corresponding target resource adjustment strategy according to the memory abnormality type of the target executor; analyzes the memory usage information of the target executor within a preset period, determines the target memory interval required for the normal operation of the task to be executed in the target executor, and creates an optimized executor based on the target resource adjustment strategy and the target memory interval; and schedules the task to be executed to the optimized executor for task execution processing. In this way, the number of executors and the resources used by the executors can be adjusted, and the multi-dimensional elastic scaling of the executor computing resources can be realized, the efficiency of computing resource allocation can be improved, and the overall resource utilization rate of the distributed cluster can be improved, and the resource fragmentation problem can be reduced.

[0125] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0126] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0127] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0128] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A distributed resource adjustment method, It is characterized in that The method is applied to a driver in a server cluster, and the method comprises: Based on the received memory usage information of each executor in the server cluster, determine a target executor with abnormal memory usage among the executors and a memory abnormality type of the target executor; wherein the memory abnormality type includes excess memory; Selecting a target resource adjustment strategy corresponding to the memory exception type of the target executor; wherein the target resource adjustment strategy corresponding to the memory surplus includes a shrinking strategy; Determining a target memory interval based on memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory interval; The method of determining a target memory interval based on the memory usage information of the target executor within a preset period, and creating an optimized executor based on the target resource adjustment strategy and the target memory interval, includes: in response to the memory exception type being the memory surplus, selecting the shrinking strategy as the target resource adjustment strategy; performing memory usage prediction based on the memory usage information of the target executor within a preset period to obtain a target memory interval; creating the optimized executor based on the shrinking strategy and the target memory interval, wherein the maximum memory value of the optimized executor is less than or equal to the minimum memory threshold value of the target executor; The tasks to be executed are dispatched to the optimized executor for task execution processing.

2. The method according to claim 1, It is characterized in that The memory exception type includes memory overflow, the resource adjustment strategy includes a capacity expansion strategy, and the target memory interval is determined based on the memory usage information of the target executor within a preset period, and the optimized executor is created based on the target resource adjustment strategy and the target memory interval, including: In response to the memory exception type being the memory overflow, selecting the capacity expansion strategy as the target resource adjustment strategy; Perform memory usage prediction based on memory usage information of the target executor within a preset period to obtain the target memory range; The optimized executor is created based on the capacity expansion strategy and the target memory range, and the maximum memory value of the optimized executor is greater than the maximum memory threshold of the target executor.

3. The method according to claim 1, It is characterized in that The step of dispatching the tasks to be executed to the optimization executor for task execution processing includes: In response to the memory exception type being a memory overflow, scheduling the to-be-executed tasks in the target executor to the optimization executor; In response to a failure in scheduling the task to be executed, rolling back the task to be executed.

4. The method according to claim 3, It is characterized in that Before rolling back the task to be executed in response to the failure of scheduling the task to be executed, the method further includes: Obtaining a task scheduling time for scheduling the to-be-executed task in the target executor to the optimized executor; In response to the task scheduling time being greater than a preset scheduling time threshold, it is determined that the scheduling of the to-be-executed task fails.

5. The method according to claim 1, It is characterized in that Before creating the optimization executor based on the scaling-in strategy and the target memory range, the method further includes: In response to the memory exception type being the excess memory, stopping allocating tasks to the target executor; In response to the completion of execution of the assigned tasks in the target executor, the target executor is closed.

6. The method according to claim 1, It is characterized in that The step of dispatching the tasks to be executed to the optimization executor for task execution processing includes: Stop allocating tasks to the target executor; In response to completion of execution of the assigned tasks in the target executor, shutting down the target executor; Allocate the tasks to be executed to the optimization executor.

7. The method according to claim 1 or 2, It is characterized in that The performing memory usage prediction based on the memory usage information of the target executor within a preset period to obtain a target memory range includes: Determining a right boundary of the target memory interval based on a maximum memory demand of the target executor within a preset period; The left boundary of the target memory interval is determined based on a minimum memory demand of the target executor within a preset period.

8. An electronic device, It is characterized in that The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, It is characterized in that When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images