A memory space allocation method and related apparatus

Abstract

The application discloses a storage space allocation method and related device. If a storage error message indicating that a first computing task fails to be executed due to insufficient storage space is acquired, a task completion parameter of a target computing task with the same task computing type as the first computing task is acquired from completed computing tasks, and a second value is estimated based on the task completion parameter and an input data amount of the first computing task. A second resource container with a storage space of the second value is created, and the first computing task is allocated to the second resource container for re-execution. Since the determined second value is more suitable for the actual required storage space when the first computing task is successfully executed, the probability of successful execution of the first computing task is improved, and the resource utilization efficiency is improved. Moreover, the storage space allocation method only needs to re-allocate the storage space of the first computing task, effectively reducing the overall time delay of data processing tasks caused by the failure of the computing task execution.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a storage space allocation method and related apparatus. Background Technology

[0002] When processing data processing tasks using distributed computing, tasks requiring massive computing power are typically broken down into multiple computational tasks. These tasks are then distributed across resource containers on multiple computing nodes for parallel execution to improve computational efficiency. If a computational task fails, an error message is sent to the management node.

[0003] In related technologies, technical personnel are usually required to process the received error messages. For example, for error messages indicating insufficient storage space, the storage space of the resource container will be increased based on the experience of the technical personnel.

[0004] However, in the above methods, the subjective experience of technicians has a significant impact on the success rate of retrying, which can easily lead to the failure of a failed computation task again, resulting in the computation task being executed repeatedly and the resource utilization efficiency being low. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a storage space allocation method and related apparatus to improve the success rate of executing computing tasks and increase resource utilization efficiency.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a storage space allocation method, the method comprising:

[0008] Obtain a storage error message sent by the first computing node for the first computing task. The first computing node is configured with a first resource container for executing the first computing task. The storage space allocated to the first resource container is a first value. The first computing task is one of multiple computing tasks obtained by decomposing data processing tasks.

[0009] In response to determining through the storage error message that the first computing task failed to execute due to insufficient storage space in the first resource container, based on the task computing type of the first computing task, the task completion parameters of the target computing task are obtained. The target computing task is a completed computing task with the same task type as the first computing task. The task completion parameters are used to identify the historical input data volume and historical storage space usage value of the target computing task.

[0010] Based on the task completion parameters and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined, wherein the second value is greater than the first value.

[0011] Create a second resource container with storage space equal to the second value, and assign the first computing task to the second resource container for re-execution.

[0012] On the other hand, embodiments of this application provide a storage space allocation device, the device comprising: an acquisition unit, a determination unit, and an allocation unit;

[0013] The acquisition unit is used to acquire a storage error message sent by the first computing node for the first computing task. The first computing node is configured with a first resource container for executing the first computing task. The storage space allocated to the first resource container is a first value. The first computing task is one of multiple computing tasks obtained by decomposing data processing tasks.

[0014] The acquisition unit is further configured to respond to determining through the storage error message that the first computing task failed to execute due to insufficient storage space in the first resource container, and based on the task computing type of the first computing task, to acquire the task completion parameters of the target computing task, wherein the target computing task is a completed computing task with the same task type as the first computing task, and the task completion parameters are used to identify the historical input data volume and historical storage space usage value of the target computing task.

[0015] The determining unit is used to determine a second value of the storage space applicable to the first computing task based on the task completion parameters and the amount of input data of the first computing task, wherein the second value is greater than the first value.

[0016] The allocation unit is used to create a second resource container with storage space of the second value, and to allocate the first computing task to the second resource container for re-execution.

[0017] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute the methods described above according to instructions in the computer program.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0021] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0022] As can be seen from the above technical solution, when processing data processing tasks, the data processing task is decomposed into multiple computing tasks, which can be processed in parallel through distributed computing. The first computing task is configured to be executed in the first resource container of the first computing node. If a storage error message indicating that the first computing task failed to execute due to insufficient storage space is obtained, the task completion parameters of the target computing task with the same task computing type as the first computing task can be obtained from the completed computing tasks. The task completion parameters indicate the amount of historical input data and the amount of historical storage space occupied when the target computing task was successfully executed. Since the target computing task is a completed computing task of the same type as the first computing task, its task completion parameters have substantial guiding significance for how to successfully execute the first computing task. Considering that the amount of input data involved in different computing tasks of the same type, i.e., the amount of data to be calculated, will be different, the amount of storage space required to successfully calculate the input data amount under the standard of successful execution of the target computing task can be estimated based on the task completion parameters and the amount of input data of the first computing task, and a second value larger than the first value can be obtained. A second resource container with a storage space of the second value is created, and the first computing task is allocated to the second resource container for re-execution. Since the determined second value is closer to the actual storage space required when the first computing task is successfully executed, it not only greatly increases the probability of the first computing task being successfully executed and effectively improves resource utilization efficiency, but also, this differentiated storage space allocation method for different computing tasks means that when the first computing task fails as described above, only the storage space of the first computing task needs to be reallocated, without affecting the normal execution of other computing tasks in the data processing task, effectively reducing the impact of computing task execution failure on the overall latency of data processing task completion. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A device interaction diagram illustrating a storage space allocation method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram illustrating an application scenario of a storage space allocation method provided in an embodiment of this application.

[0026] Figure 3 A flowchart illustrating a storage space allocation method provided in an embodiment of this application;

[0027] Figure 4a A flowchart illustrating a memory allocation mechanism provided in an embodiment of this application;

[0028] Figure 4b This is a flowchart illustrating a process for updating memory parameters.

[0029] Figure 5 A schematic diagram illustrating a process for individually updating memory parameters, provided as an embodiment of this application;

[0030] Figure 6 A flowchart illustrating an example of an application storage space allocation method provided in this application embodiment;

[0031] Figure 7 A schematic diagram illustrating the process of a computing node reporting messages, provided as an embodiment of this application;

[0032] Figure 8 A flowchart illustrating the process of a management node parsing messages is provided for an embodiment of this application;

[0033] Figure 9 A flowchart illustrating an application scenario of a storage space allocation method provided in this application embodiment;

[0034] Figure 10 This is a schematic diagram of a storage space allocation device provided in an embodiment of this application;

[0035] Figure 11 A structural diagram of a terminal device provided in an embodiment of this application;

[0036] Figure 12 This is a structural diagram of a server provided in an embodiment of this application. Detailed Implementation

[0037] The embodiments of this application will now be described with reference to the accompanying drawings.

[0038] To improve the efficiency of solving complex tasks, distributed computing is typically used to decompose data processing tasks into multiple computational tasks (Tasks). A management node (Master) then distributes these tasks to different computational nodes (Workers). The computational nodes execute the tasks, while the management node manages and controls them. Tasks assigned to the same node share its resources. Each task runs within a resource container to ensure environmental isolation during execution. Executing computational tasks requires storage space. In related technologies, a standardized storage space requirement for each task is typically determined first. Multiple resource containers are then created based on this standardized value, and each task is assigned to a different resource container.

[0039] If the storage space required by a computing task exceeds the storage space provided by its resource container during the execution of the computing task, the task will fail due to insufficient storage space, and the resource container of the computing node will report an error message indicating that the task has failed.

[0040] As mentioned above, a uniform value is used to identify the storage space of each resource container. In related technologies, technicians typically need to adjust this uniform value based on their experience or simple rules, re-apply for resource containers according to the adjusted uniform value, and then assign each computing task to the newly created resource container for re-execution. For example, a linear increment rule increases the uniform value by a fixed amount each time, while an exponential increment rule doubles the uniform value each time.

[0041] Therefore, it is evident that whether based on subjective experience or simple rules, the updated unified values ​​do not consider the characteristics of the failed computation tasks themselves, lack specificity, and are difficult to match the actual storage space required by the failed computation tasks. This results in a high probability of failure when re-executing the computation tasks, requiring more value adjustments and re-executions, leading to low computational resource utilization efficiency. Based on this, embodiments of this application provide a storage space allocation method and related apparatus. After obtaining a storage error message for a first computation task, the method obtains the task completion parameters of a target computation task with the same task type as the first computation task. These task completion parameters identify the historical input data volume and historical storage space usage of the target computation task. By using the task completion parameters of completed computation tasks of the same task type as a reference, combined with the input data volume of the first computation task, the method determines how much storage space the first computation task needs to successfully compute the input data volume, obtaining a second value that more closely approximates the actual storage space required by the first computation task. Thus, the first computation task is allocated to a second resource container with the second value of storage space for re-execution, resulting in a higher success rate, significantly reduced retries, and improved computational resource utilization efficiency.

[0042] The storage space allocation method provided in this application can be implemented using a computer device, which can be a terminal device or a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, in-vehicle terminals, and extended reality (XR) devices. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0043] To facilitate understanding of the storage space allocation method provided in this application embodiment, the following example uses a server as the execution subject of the storage space allocation method and a management node as an example to illustrate the application scenario of the storage space allocation method.

[0044] See Figure 1 This figure is a schematic diagram of device interaction for a storage space allocation method provided in an embodiment of this application.

[0045] The management node 110 can be an independent server used to manage and control the computing nodes 120, 130 and 140. The computing nodes are terminal devices or servers that can perform computing tasks and have computing capabilities. The management node 110 communicates with each computing node through a communication network. The management node is used to receive the amount of input data from the computing nodes, store error messages, and send values ​​to the computing nodes to indicate the size of the resource container storage space.

[0046] The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, private network, or any combination of virtual private network. In some embodiments, custom or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0047] The following example uses a management node and the first and second computing nodes it controls to illustrate the storage space allocation method provided in this application.

[0048] See Figure 2 This figure is a schematic diagram of an application scenario for a storage space allocation method provided in an embodiment of this application, wherein the execution subject of the storage space allocation method is the management node 230. Figure 2As shown, the application scenario includes a first computing node 220, a second computing node 210, and a management node 230. The second computing node 210 and the management node 230 can communicate through a communication network, and the first computing node 210 and the management node 230 can also communicate through a communication network.

[0049] The management node 230 can be an independent server used to manage and control the various computing nodes, while the second computing node 210 and the first computing node 220 can be servers used to execute specific computing tasks. First, the management node 230 decomposes the data processing task into multiple computing tasks, such as a target computing task, a first computing task, a second computing task, etc., where the target computing task and the first computing task have the same computing type. Each computing task is assigned to a resource container on a different computing node for execution. For example, the target computing task is assigned to the fourth resource container of the second computing node 210, and the first computing task is assigned to the first resource container of the first computing node 220. The storage space size of the first resource container is a first value: 2GB (Gigabyte).

[0050] During the parallel execution of various computing tasks, the first computing task fails due to insufficient storage space in the first resource container. The first computing node 220 sends a storage error message for the first computing task to the management node 230. Based on the task computation type of the first computing task, the management node 230 obtains the task completion parameters of the target computing task from the second computing node 210. These parameters identify the historical input data volume and historical storage space usage of the target computing task. Using these completion parameters and the input data volume of the first computing task sent by the first computing node 220, the management node 230 can estimate how much storage space is needed to successfully compute the input data volume, based on the criteria for successful execution of the target computing task. For example, if a second storage space value of 3GB is determined to be suitable for the first computing task, a second resource container with 3GB of storage space is created on the first computing node 220. The first computing task is then re-executed in this second resource container, which better matches the actual storage space requirements of the first computing task, increasing the probability of successful re-execution and reducing the waste of computing resources. Furthermore, the aforementioned storage space allocation process only requires reallocating storage space for the first computing task, without needing to re-execute completed computing tasks such as the target computing task. Subsequent computing tasks can continue to be executed in an orderly manner in other resource containers, effectively reducing the impact of computing task execution failures on the overall latency of data processing task completion.

[0051] It should be noted that the above application scenarios are merely examples, and the storage space allocation method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0052] The following describes in detail a storage space allocation method provided in this application through method embodiments.

[0053] See Figure 3 This figure is a schematic flowchart of a storage space allocation method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of the storage space allocation method, and the server as a management node as an example. Figure 3 As shown, the storage space allocation method includes the following steps:

[0054] S301: Obtain the storage error message sent by the first computing node for the first computing task.

[0055] Data processing tasks require significant computing power to complete. To improve efficiency, distributed computing breaks down these tasks into multiple computational tasks. For example, tasks can be categorized and broken down according to their computational type, and each task can be assigned to multiple computing nodes for parallel execution. Finally, the results from all computational tasks are combined to obtain the final result of the data processing task. These different computing nodes can be computer devices with computing capabilities located in different geographical locations, connected through a communication network to achieve the computational goals of the data processing task.

[0056] A distributed computing engine is a distributed computing system that executes computing tasks. It comprises a computing cluster consisting of multiple node devices. The management node is the central node of the distributed computing engine, responsible for managing and controlling the other node devices in the computing cluster, such as task allocation, resource scheduling, and state management. Compute nodes are the node devices that specifically execute computing tasks. Each compute node can be configured with at least one resource container. A resource container is a lightweight, standardized virtualization environment used to encapsulate the applications and dependencies required to execute computing tasks, allowing different computing tasks to run independently in different resource containers on the same compute node. It should be noted that the management node can allocate multiple computing tasks to the same compute node, sharing the node's computing resources.

[0057] The computing task is assigned to a resource container with a storage space size corresponding to the specified value. The specific value of the storage space size can be determined by the management node. For example, if the management node indicates that the storage space required for computing task A is 2GB, then computing task A will be assigned to a 2GB resource container.

[0058] If the storage space required to complete the computing task exceeds the storage space of its resource container, the computing task will fail. In this case, it is necessary to reallocate a larger storage resource container for the computing task to be re-executed. The storage space allocation method of this application embodiment will be described in detail below, taking the allocation of storage space for the first computing task as an example.

[0059] The first computing task is one of several computing tasks obtained by decomposing data processing tasks, and the first computing node is one of several computing nodes used to execute the computing task. In at least one resource container configured on the first computing node, the first resource container is used to execute the first computing task and is allocated storage space of a first value, i.e., the storage capacity of the first resource container is the allocated storage space size. This first value is used to identify the storage capacity of the first resource container. A storage error message is a message sent to the management node corresponding to the computing node where the computing task is located, triggered by the computing task failing due to insufficient required storage space. If the actual storage space required to complete the first computing task is greater than the first value, the first computing task will fail due to insufficient storage space in the first resource container. The first computing node will report this situation, and the management node will obtain the storage error message sent by the first computing node, which indicates that the first computing task failed due to insufficient storage space.

[0060] The insufficient storage space for the first computing task may be due to two reasons: data skew and data bloat.

[0061] The first scenario is data skew. During the parallel execution of various computing tasks, the uneven distribution of input data obtained from the decomposition of data processing tasks results in a significantly higher input data volume for the first computing task compared to other computing tasks, leading to data skew. In this case, the storage space of the first resource container is insufficient to handle the massive amount of input data.

[0062] The second scenario is data bloat. The computational method used to perform the first computational task generates a large amount of intermediate data, such as cache files and log records, causing a surge in the storage space required for the first computation, resulting in data bloat. The storage space of the first resource container becomes insufficient to accommodate the massive amount of intermediate data.

[0063] In one possible implementation, the storage space identified by the error message can be at least one of the memory space allocated to the first resource container or a buffer space. The memory space is an area allocated to the resource container for storing temporary data for computational tasks. The memory space is primarily used to load programs and their dependencies, and store variable values, stack information, etc. The buffer space is a temporary data storage area used to coordinate data transfer between devices or processes of different speeds, balancing the rate differences between high-speed and low-speed devices.

[0064] It should be noted that the content of the storage error message can be used solely to indicate that the first computing task failed due to insufficient storage space. That is, regardless of the type of storage space shortage causing the failure, the storage error message only needs to allow the management node to identify that the failure was due to insufficient storage space in the first resource container, thus clearly indicating that the management node needs to allocate a resource container with larger storage space to re-execute the first computing task. The content of the storage error message can also be further refined based on the specific storage space type, enabling the management node to identify which type of storage space shortage caused the first computing task to fail.

[0065] For example, when insufficient memory prevents the storage of intermediate data generated by a computation task, further computation cannot be performed, causing the task to fail. The management node will receive a memory out-of-memory (OOM) error message. Similarly, in I / O-intensive computation tasks, if the amount of data input to the buffer space exceeds its pre-allocated space size, causing the excess to overwrite adjacent storage space, the task will fail, and the management node will receive a buffer overflow error message. Therefore, the management node can adopt targeted storage space allocation strategies based on the type of storage space, improving the accuracy of storage space allocation. If the management node receives an OutOfMemory error, it will allocate a second resource container with a larger memory space for the first computation task; if the management node receives a BufferOverflow error, it will allocate a second resource container with a larger buffer space for the first computation task.

[0066] Therefore, both insufficient memory space and buffer overflow will cause the compute node to report a storage error message with specified content. These storage error messages can be identified by the management node and processed by reallocating storage space, which improves the applicability of automated retry compute tasks.

[0067] S302: Based on the task calculation type of the first calculation task, obtain the task completion parameters of the target calculation task.

[0068] Each computational task involves at least one computational method, such as averaging, summing, finding maximum / minimum values, sorting, searching, and grouping. Completing a computational task requires execution in the prescribed order of its computational methods. Therefore, the combination of computational methods is a substantial factor affecting the storage space required for a computational task. Based on this, each computational task can be categorized into a task computation type according to its computational method. The task computation type identifies the combination of computational methods for a computational task; different task computation types correspond to different combinations of computational methods, and their impact on the required storage space varies. In other words, for different computational tasks of the same computational type, because they all possess the same combination of computational methods, the impact of each task's computational method on the required storage space is similar when executing these tasks. Therefore, when processing the same amount of input data, the actual difference in the required storage space is relatively small.

[0069] Taking a data processing task to calculate the sales revenue of different mobile phone brands as an example, completing this task involves reading data, classifying mobile phone brands, statistically analyzing mobile phone models, and summing sales revenue. This data processing task can be broken down into two computational tasks: Task A and Task B. Task A sums the sales revenue of brand A, and Task B sums the sales revenue of brand B. Both Task A and Task B involve summation operations. Therefore, based on the storage space required to successfully complete Task A, we can infer the impact of the summation operation on the required storage space in Task B.

[0070] The target computation task is a completed computation task of the same type as the first computation task. Based on this, the task completion parameters of the target computation task can be obtained, thereby estimating the impact of the computation methods involved in completing the first computation task on the required storage space. These task completion parameters are used to identify the historical input data volume and historical storage space usage of the target computation task. The historical input data volume is the total amount of raw data that the completed computation task needs to process, and the historical storage space usage is the actual storage space size of the resource container occupied by the completed computation task. For example, the historical storage space usage could be the peak storage space size of the resource container actually occupied by the completed computation task, used to represent the maximum storage space required to complete the computation task.

[0071] It should be noted that any completed computation task with the same computation type as the first computation task can be used as the target computation task. For example, the target computation task and the first computation task can be two computation tasks derived from the same data processing task. The target computation task can also be a computation task from other data processing tasks; there are no restrictions here.

[0072] S303: Determine a second value for the storage space applicable to the first computing task based on the task completion parameters and the amount of input data for the first computing task.

[0073] The second resource container is a resource container reallocated for the first computing task. The second resource container is used to re-execute the first computing task, and its allocated storage space size is a second value. That is, the storage capacity of the second resource container is the size of the allocated storage space, and this second value is used to identify the storage capacity of the second resource container. The input data volume is the total amount of raw data to be processed when executing the computing task. For example, the input data volume could be the size of the file to be processed read by the computing task. When the data processing task is divided into multiple stages, the input data volume could also be the size of the output data from the previous stage of the computing task.

[0074] Computational tasks of the same type may involve different amounts of input data. For example, two calculation tasks for calculating sales revenue may involve different quantities of products. Therefore, when the task completion parameters of the target computational task are substantially instructive for the successful execution of the first computational task, the amount of storage space required to successfully calculate that amount of input data, based on the input data volume of the first computational task, can be estimated. This second value, greater than the first value, determines the appropriate storage space for the first computational task. For instance, using the task completion parameters as a reference, the correspondence between data volume and storage space in the successful completion of similar tasks can be obtained. Based on this correspondence, the specific storage space required to successfully complete the first computational task can be estimated according to its input data volume.

[0075] S304: Create a second resource container with a storage space of the second value, and assign the first computing task to the second resource container for re-execution.

[0076] After determining the second value, storage space is reallocated to the first computing task according to the second value. A second resource container with storage space equal to the second value can be created, and the first computing task can be re-executed within the second resource container. The second resource container is a newly allocated resource container for re-executing the first computing task. The virtualization environment of the second resource container is the same as that of the first resource container, but the storage space of the second resource container is larger than that of the first resource container.

[0077] In one possible implementation, a second resource container with a storage space of a second value can be created by requesting the creation of a second resource container with a storage space of a second value on a first computing node, or by requesting the creation of a second resource container with a storage space of a second value on a second computing node different from the first computing node.

[0078] For example, in this embodiment, if the first computing node fails to execute due to insufficient storage space in the first resource container, a second resource container with larger storage space can be created and requested on the first computing node, and the computing task can be re-executed in the second resource container. When the available storage space on the first computing node is insufficient to request a larger resource container, a second resource container can also be created on other computing nodes with more available storage space. This not only achieves load balancing between different computing nodes but also improves the efficiency of storage space allocation and reduces the latency of completing data processing tasks. Therefore, the requested second resource container can be configured on different computing nodes in the computing cluster, improving the flexibility of storage space allocation.

[0079] Since the second value is closer to the actual storage space required by the first computing task, it is more accurate than human experience and simple rules. It will not result in a low probability of successful execution due to the storage space of the second resource container being too small, requiring multiple retries, nor will it waste storage space due to the storage space of the second resource container being too large.

[0080] Furthermore, in related technologies, if a data processing task involves multiple stages, and each stage is decomposed into multiple computing tasks, if the computing task in one stage fails, a unified value needs to be adjusted upwards. This unified value is used to indicate the storage space size of the resource container for each stage's computing task. Therefore, the resource containers for each stage's computing task need to be reallocated, and each computing task needs to be re-executed starting from the first stage, resulting in an overall delay in completing the data processing task.

[0081] The following is through Figure 4a , Figure 4b and Figure 5 The two storage space allocation methods are described in detail to demonstrate the specific improvements made to the relevant technology by the embodiments of this application.

[0082] See Figure 4a The figure is a flowchart of a memory allocation mechanism provided in an embodiment of this application. The management node receives a memory parameter of size 2GB, which is used to indicate the memory space size of the resource container. The management node applies for and creates a resource container with a memory space of 2GB on the computing node for each computing task according to the memory parameter, and allocates each computing task to the corresponding resource container for execution.

[0083] like Figure 4bAs shown in the related technologies, when the resource container indicated by the memory parameter cannot meet the memory space required by the computing task, the management node receives a message indicating that the computing task has failed. It then increases the size of the memory parameter and reallocates new resource containers to each computing task according to the increased memory parameter. Although failed computing tasks are reassigned to resource containers with larger memory spaces for execution during this process, it is important to emphasize that the memory parameter in the related technologies controls the memory size of all resource containers. This means that if any computing task fails, causing the memory parameter to be increased, all resource containers controlled by that memory parameter need to be replaced with larger memory containers. Consequently, the same batch of computing tasks derived from the same data processing task need to be reassigned to new resource containers for re-execution, and even successfully executed computing tasks need to be re-executed, significantly wasting the resources of the computing node and reducing the execution efficiency of data processing tasks.

[0084] In the embodiments of this application, see Figure 5 Memory parameters 1, 2, 3, and 4 are used to indicate the memory size of different resource containers. For example, in the initial stage of memory allocation, all four memory parameters indicate that the corresponding resource container's memory size is 2GB. Taking the computation task corresponding to memory parameter 4 as an example, this computation task requires more memory than the 2GB indicated by memory parameter 4, causing execution failure. Since different memory parameters indicate the memory size of their respective resource containers, the management node, after receiving a storage error message, does not need to adjust all memory parameters; it only needs to increase memory parameter 4, such as from 2GB to 4GB. The failed computation task is then reassigned to a 4GB resource container for re-execution. The retry process does not require other successfully completed computation tasks to be executed again, improving resource utilization and the execution efficiency of data processing tasks.

[0085] As can be seen from the above technical solution, when processing data processing tasks, the data processing task is decomposed into multiple computing tasks, which can be processed in parallel through distributed computing. The first computing task is configured to be executed in the first resource container of the first computing node. If a storage error message indicating that the first computing task failed to execute due to insufficient storage space is obtained, the task completion parameters of the target computing task with the same task computing type as the first computing task can be obtained from the completed computing tasks. The task completion parameters indicate the amount of historical input data and the amount of historical storage space occupied when the target computing task was successfully executed. Since the target computing task is a completed computing task of the same type as the first computing task, its task completion parameters have substantial guiding significance for how to successfully execute the first computing task. Considering that the amount of input data involved in different computing tasks of the same type, i.e., the amount of data to be calculated, will be different, the amount of storage space required to successfully calculate the input data amount under the standard of successful execution of the target computing task can be estimated based on the task completion parameters and the amount of input data of the first computing task, and a second value larger than the first value can be obtained. A second resource container with a storage space of the second value is created, and the first computing task is allocated to the second resource container for re-execution. Since the determined second value is closer to the actual storage space required when the first computing task is successfully executed, it not only greatly increases the probability of the first computing task being successfully executed and effectively improves resource utilization efficiency, but also, this differentiated storage space allocation method for different computing tasks means that when the first computing task fails as described above, only the storage space of the first computing task needs to be reallocated, without affecting the normal execution of other computing tasks in the data processing task, effectively reducing the impact of computing task execution failure on the overall latency of data processing task completion.

[0086] This application does not specifically limit how the second value is determined by the task completion parameters and the amount of input data of the first calculation task. The following is an example of one method.

[0087] An example of establishing a linear relationship between data volume and storage space:

[0088] First, by identifying the historical input data volume and historical storage space usage values ​​of the task completion parameters, we can determine the linear relationship between the data volume and storage space for the target computation task.

[0089] A linear relationship represents a fixed proportion between two variables. In this context, a linear relationship between data volume and storage space indicates that there is a fixed proportion between the storage space required to complete the target computation task and the amount of input data. This linear relationship can be obtained by determining the ratio between the historical storage space occupied by the target computation task and the historical input data volume. For example, this linear relationship can be determined using the following formula:

[0090] beta=MemUsage / DataInput (Formula 1)

[0091] Wherein, DataInput is the amount of historical input data for task completion parameter identification, MemUsage is the historical storage space occupied by task completion parameter identification, and beta is a proportional parameter used to indicate the linear relationship.

[0092] As mentioned above, for computational tasks of the same type, the impact of the computation method on the required storage space is similar. Therefore, this impact can be quantified by examining the linear relationship between the data volume and storage space for the target computational task, and the relationship between the storage space required to complete other computational tasks of the same type and the amount of input data can be inferred.

[0093] When the amount of input data changes, the storage space required for the same type of computing task changes proportionally to the change in the amount of input data. For example, for computing tasks of the same type, for every 1GB increase in input data, the required storage space needs to increase by 2GB. This pattern demonstrates the linear relationship between data volume and storage space for this type of computing task.

[0094] In estimating the storage space required to retry the first computational task, a second value for the applicable storage space can be determined based on this linear relationship and the amount of input data for the first computational task. For example, the parameter beta obtained through the above formula can be used to obtain the second value through the following formula:

[0095] mem1=beta×taskDataInput (Formula 2)

[0096] Where mem1 is the second value, beta is the scale parameter used to indicate the linear relationship, and taskDataInput is the amount of input data for the first calculation task.

[0097] Therefore, by obtaining the linear relationship between data volume and storage space for the target computing task through task completion parameters, the ratio between the storage space required to complete the same type of computing task and the amount of input data can be determined. Different computing tasks have different amounts of input data due to data skew. Therefore, by combining the input data volume of the first computing task with this linear relationship, the storage space required for the first computing task can be estimated. The linear storage relationship of the same type of completed tasks can be transferred to the first computing task, improving the accuracy of the second value. The first computing task can be allocated to a resource container with storage space that is more in line with actual needs, thus increasing the success rate of successfully executing the first computing task.

[0098] An example of determining the exponentially non-linear relationship between data volume and storage space:

[0099] As can be seen from the foregoing, the reasons for insufficient storage space in computing tasks are not only data skew, but also the surge in required storage space caused by the generation of a large amount of intermediate data by computing tasks. Considering this data expansion phenomenon, this application embodiment also provides a specific implementation method for simulating this expansion phenomenon through an exponential nonlinear relationship to further determine the second value.

[0100] By analyzing historical input data volume and historical storage space usage, we can determine the exponential nonlinear relationship between the target computation task and the data volume / storage space.

[0101] Unlike linear relationships, exponential nonlinear relationships are based on exponential functions. The exponential nonlinear relationship between data volume and storage space indicates the target computation task. As the amount of input data changes, the storage space required to complete the target computation task changes exponentially. For example, for the same type of computation task, for every 1KB (kilobyte) increase in input data, the required storage space increases by 1GB, where 1GB = (1K × 1K × 1K)B. The increase in storage space no longer changes linearly with the amount of input data; instead, the increase in required storage space is proportional to the cube of the amount of input data. This pattern demonstrates the exponential nonlinear relationship between data volume and storage space for this type of computation task.

[0102] Therefore, the exponential linear relationship can be obtained by determining the ratio between the historical storage space occupied by the target computation task and the exponential change in the amount of historical input data. For example, the exponential nonlinear relationship can be determined using the following formula:

[0103] alpha = MemUsage / (DataInput) n ) (Formula 3)

[0104] Where alpha is a scaling parameter indicating the exponential nonlinear relationship, MemUsage is the historical storage space occupied by the task completion parameter identifier, DataInput is the historical input data volume of the task completion parameter identifier, and n is an integer greater than 1. The larger n is, the more significant the data inflation phenomenon, and the more the storage space required to complete the target calculation task changes with the amount of input data. n This is used to identify the exponential change in the amount of historical input data for the target computing task. The embodiments of this application do not limit the specific value of n; n can be set by those skilled in the art based on the task computing type of the target computing task.

[0105] The proportionality constant alpha determined by Formula 3 can simulate the ratio of the storage space required to complete the target computation task to the nth power of the input data volume in the case of exponential data expansion based on n.

[0106] It should be noted that the proportionality constant alpha can be determined in two ways:

[0107] In the first method, n can be used to identify a specified exponent. That is, by substituting n into formula 3, an alpha is obtained, thereby simulating the exponential data expansion phenomenon based on n through an alpha.

[0108] In the second method, n can also be used to indicate the upper limit of the exponent in the data inflation phenomenon. That is, the exponents involved in the data inflation phenomenon include 2, 3, ..., n. By substituting multiple exponents into Formula 3, multiple alphas can be obtained, thus simulating data inflation phenomena of different exponential levels in the target calculation task through multiple alphas. For example, if n is 3, then the exponents involved in the data inflation phenomenon include 2 and 3. Substituting n=2 and n=3 into Formula 3 respectively, we obtain the scaling factor alpha1 for simulating quadratic level data inflation and the scaling factor alpha2 for simulating cubic level data inflation.

[0109] In one possible implementation, the exponential nonlinear relationship is a second-order exponential nonlinear relationship. That is, in the data inflation phenomenon, the storage space required to complete the target computation task is proportional to the square of the input data volume, and this proportion is used to indicate the exponential nonlinear relationship of the target computation task.

[0110] In distributed computing tasks, the storage space occupied by intermediate data often does not exceed the square of the input data. In other words, the second-order exponential nonlinear relationship can cover most data inflation phenomena. Thus, by using a nonlinear relationship to represent most data inflation phenomena, the complexity of exponential nonlinear relationships is simplified, and the computational cost of determining the second value is reduced.

[0111] Given a fixed amount of input data, by determining the linear relationship between the target computation task and the data volume / storage space, we can estimate the storage space required to process the input data under this linear storage relationship. By determining the exponential nonlinear relationship between the target computation task and the data volume / storage space, we can estimate the additional storage space required for intermediate data during data expansion. Since the target computation task and the first computation task belong to the same computation type, the two storage relationships corresponding to the successfully completed target computation task can be migrated to the first computation task. That is, the linear relationship is used to simulate the linear storage relationship of the first computation task, and the exponential nonlinear relationship is used to simulate the data expansion phenomenon of the first computation task. Therefore, based on the linear relationship, the exponential nonlinear relationship, and the input data volume of the first computation task, a second value for the storage space applicable to the first computation task can be determined.

[0112] Specifically, based on the amount of input data for the first computation task and the linear relationship, a first storage space is determined to complete the first computation task. This first storage space is the storage space that needs to be reserved for this linear storage relationship. For example, the first storage space can be obtained using the following formula:

[0113] submem1=beta×taskDataInput (Formula 4)

[0114] Wherein, submem1 is the first storage space, beta is the proportional parameter used to indicate the linear relationship, beta can be determined by the historical input data volume and historical storage space usage value of the task completion parameter identifier, and taskDataInput is the input data volume of the first calculation task.

[0115] Then, based on the amount of input data for the first computational task and the exponential nonlinear relationship, a second storage space is determined to complete the first computational task. This second storage space is reserved for the intermediate data generated by the data expansion phenomenon. For example, the second storage space can be obtained using the following formula:

[0116] submem2=alpha× (taskDataInput n ) (Formula 5)

[0117] Where submem2 is the second storage space, taskDataInput is the amount of input data for the first computation task, and n is an integer greater than 1 used to indicate the degree of data bloat in the first computation task. nThis is used to identify the exponential change in the amount of input data for the first computational task. If multiple alpha values ​​are obtained from Equation 3, then these multiple alpha values ​​and their corresponding exponents are substituted into Equation 5 to obtain multiple submem2 values, in order to estimate the storage space required for different exponential data expansion phenomena.

[0118] Finally, based on the first and second storage spaces, the total storage space required to complete the first computational task, considering both linear storage relationships and data bloat, is determined, thus establishing the second value. For example, if n is used to identify a specified exponential level, the second value can be obtained using the following formula:

[0119] mem1=submem1+submem2=alpha×taskDataInput n +beta×

[0120] taskDataInput(Formula 6)

[0121] Where mem1 is the second value, submem1 is the first storage space, submem2 is the second storage space, alpha is a scaling parameter used to indicate the exponential nonlinear relationship, taskDataInput is the amount of input data for the first computational task, and n is an integer greater than 1 used to indicate the severity of the data inflation phenomenon in the first computational task. n beta is used to identify the exponential change in the amount of input data for the first computational task, and is a proportional parameter used to indicate this linear relationship.

[0122] As can be seen from Formula 6, in the process of estimating the second value based on the proportional parameters alpha and beta, the exponent corresponding to the input data volume is consistent with the exponent of the aforementioned proportional parameters. For example, the input data volume after alpha is nth power, and the input data volume after beta is 1st power. This is to transform the linear relationship and the exponential nonlinear relationship into the same dimension, so as to uniformly measure the impact of linear storage relationship and data expansion phenomenon on the second value.

[0123] Therefore, by using the task completion parameters, an exponential nonlinear relationship between the data volume and storage space of the target computing task can be obtained. This exponential nonlinear relationship can be used to simulate the data expansion phenomenon of the target computing task, thereby determining a fixed ratio between the surge in intermediate data and the exponential change in the input data volume. As a first computing task of the same type as the target computing task, this nonlinear relationship and the input data volume can be used to determine how much additional storage space needs to be reserved for intermediate data in order to successfully complete the first computing task. Thus, based on the storage space determined by the linear storage relationship, the storage space requirements of the data expansion phenomenon are further considered, improving the accuracy of the second value. This allows for the creation of a resource container that better fits the storage space requirements of the first computing task based on the second value.

[0124] An example of determining the linear mean relationship between data volume and storage space:

[0125] To improve the robustness and applicability of the second value, embodiments of this application also provide a specific implementation method for determining the second value based on multiple task completion parameters.

[0126] Based on the task calculation type of the first calculation task, obtain the task completion parameters of each historical calculation task in the target calculation task set.

[0127] The target computing task set includes multiple historical computing tasks. Historical computing tasks are completed computing tasks with the same task type as the first computing task. The task completion parameters of historical computing tasks are used to identify the historical input data and historical storage space usage values ​​of historical computing tasks.

[0128] After determining the task computation type of the first computation task, historical computation tasks with the same task type as the first computation task can be identified from multiple completed computation tasks obtained from the decomposition of the data processing task. Historical computation tasks with the same task type as the first computation task can also be identified from other computation tasks. This allows for the determination of a target computation task set that includes multiple historical computation tasks. The task completion parameters corresponding to each historical computation task can then be obtained. Based on these multiple task completion parameters, the amount of input data and the corresponding storage space requirements for completing multiple tasks of this type can be determined.

[0129] By identifying the historical input data volume and historical storage space usage values ​​through the task completion parameters of historical computing tasks, the linear mean relationship between data volume and storage space for the target computing task set is determined.

[0130] The average input data volume of the target computing task set can be determined by using multiple historical input data volumes identified by multiple task completion parameters. This average input data volume reflects the average size of the raw data that each historical computing task needs to process. The average storage space usage value of the target computing task set can be determined by using multiple historical space usage values ​​identified by multiple task completion parameters. This average storage space usage value reflects the average size of storage space required to complete the computing type of the task under the condition of processing the average input data volume.

[0131] A linear mean relationship represents a fixed proportion between two mean variables. In the context of a target computational task set, the linear mean relationship between data volume and storage space indicates that the average storage space required by each historical computational task in the target task set is proportional to the average input volume of each historical computational task. This linear mean relationship can be obtained by determining this proportion. For example, the linear mean relationship can be determined using the following formula:

[0132] beta=avgMemUsage / avgDataInput (Formula 7)

[0133] Where avgDataInput is the average amount of input data for the target computing task set, avgMemUsage is the average storage space occupied by the target computing task set, and beta is a proportional parameter used to indicate the linear mean relationship.

[0134] Therefore, the second value can be determined based on the linear mean relationship and the amount of input data for the first calculation task. This process can be referred to the aforementioned steps and will not be repeated here.

[0135] This linear mean relationship is determined by multiple task completion parameters from multiple historical computing tasks. It can reflect the average level of storage space requirements for completing the same type of task. Therefore, based on this linear mean relationship, the influence of individual differences in computing tasks on the estimation of the second value can be reduced. It can more accurately reflect the relationship between the storage space occupied by the computing type of task and the amount of input data, thus improving the robustness and applicability of the second value.

[0136] An example of determining the exponentially nonlinear mean relationship between data volume and storage space:

[0137] It should be noted that, based on the task calculation type of the first calculation task, multiple task completion parameters of each historical calculation task in the target calculation task set can be obtained. By using the historical input data volume and historical storage space usage value identified by the task completion parameters of the historical calculation tasks, the exponential nonlinear mean relationship of the target calculation task set can be determined.

[0138] This means that the average input and average storage space usage of the target computation task set can be determined based on multiple task completion parameters. The ratio between the average storage space usage and the exponential change in the average input can then be determined using these values, thus yielding an exponential nonlinear mean relationship. For example, this exponential nonlinear mean relationship can be determined using the following formula:

[0139] alpha=avgMemUsage / (avgDataInput n ) (Formula 8)

[0140] Where DataInput is the average input data volume of the target computing task set, avgMemUsage is the average storage space usage of the target computing task set, n is an integer greater than 1 used to indicate the significance of the data inflation phenomenon in the first computing task, and alpha is a proportional parameter used to indicate this exponential nonlinear mean relationship. Referring to Formula 3, one alpha can be determined by n, or multiple indices involved in the data inflation phenomenon can be determined by using n as an upper limit of the exponent. Substituting these multiple indices into Formula 8 yields multiple alphas.

[0141] Therefore, the exponential nonlinear mean relationship determined by the parameters of multiple tasks can reflect the average level of data expansion when completing the same type of task. Thus, based on this linear mean relationship, the impact of individual differences in computational tasks on the storage space required to estimate the surge in intermediate data can be reduced, thereby improving the robustness and applicability of the second value.

[0142] Implementation example of allocating storage space based on the number of retries:

[0143] In distributed computing, as the number of times a computation task is re-executed increases, the additional computing resources consumed also increase, while the overall task execution efficiency decreases. For example, if a data processing task is divided into multiple stages, and the first computation task in the first stage fails multiple times, subsequent stages that depend on the result of the first computation task will be unable to proceed, leading to a continuous increase in the completion time of the data processing task. Based on this, this application proposes a storage space allocation method to reduce the number of retries.

[0144] First, the retry weight can be determined based on the number of times the first computation task is re-executed.

[0145] The retry margin is a dynamically adjusted weight based on the number of retries. The retry margin changes as the first computation task is re-executed, with a larger margin for each subsequent retry. To improve the success rate of the next execution of the first computation task, the retry margin can be adjusted to increase the storage space allocated to the resource container for that next execution. In other words, the more times the first computation task is re-executed, the larger the storage capacity of the resource container determined by the retry margin, thus reducing the probability of failure in subsequent retryes and consequently decreasing the number of retries required for the first computation task.

[0146] For example, if the retry count weight is used as a compensation value for storage space, then the retry count weight can be determined by the following formula:

[0147] margin = const1 × (tryNum 2 ) (Formula 9)

[0148] In this formula, margin represents the weight of the retry count, const1 is the first fixed parameter, which can be set by those skilled in the art according to the task calculation type, and tryNum is the number of times the first calculation task is re-executed. In this formula, the weight of the retry count increases as the number of re-executions of the first calculation task increases.

[0149] Based on the linear relationship, the weight of the number of retries, and the amount of input data for the first computation task, a second value for the storage space applicable to the first computation task is determined.

[0150] As mentioned above, based on the linear relationship between the data volume and storage space of the target computing task and the input data volume of the first computing task, the linear storage relationship of the target computing task can be transferred to the first computing task, thereby estimating the storage space required by the first computing task. Furthermore, to reduce the number of retries for the first computing task and increase the success rate of future retries, the second value can be adjusted using the retry weight determined by the number of retries, allocating more storage space for the next execution of the first computing task.

[0151] The following example illustrates how to determine the second value based on the weight of the number of retries.

[0152] If the retry count weight is used as a compensation value for storage space, and the retry count weight increases as the number of times the first computation task is re-executed increases, then the second value can be determined by the following formula:

[0153] mem1=beta×taskDataInput+margin(Formula 10)

[0154] Where mem1 is the second value, beta is the proportional parameter indicating the linear relationship, taskDataInput is the input data volume of the first computation task, and margin is the weight of the retry count. It can be seen that in this formula, as the number of times the first computation task is re-executed increases, the weight of the retry count also increases. Based on the determined storage space corresponding to the linear storage relationship, the second value increases with the increase of the retry count weight, thus allocating more resource container storage space for the next retry of the first computation task, reducing the probability of the first computation task failing in the next retry.

[0155] Therefore, by determining the weight of the retry count and calculating the second value with the goal of expanding storage space, a larger second value can be determined based on the original allocated storage space. This allows the first computational task to be allocated to a larger resource container, increasing the probability of success in the next execution of the first computational task. Simultaneously, as the number of retries increases, the storage space of the resource container becomes larger, further increasing the probability of success, reducing the likelihood of retrying the first computational task, minimizing resource waste, and improving the overall task execution efficiency.

[0156] An example of allocating storage space using multiple undetermined values:

[0157] In one possible implementation, if the first computation task is executed for the first time through the third resource container and fails due to insufficient storage space, the third resource container is allocated storage space of a third value. Then, the first pending value can be determined based on the third value and the number of times the first computation task is re-executed.

[0158] The third resource container is a resource container allocated for the initial execution of the first computing task. The allocated storage space size for the third resource container is a third value, meaning the storage capacity of the third resource container is the size of the allocated storage space. This third value identifies the storage capacity of the third resource container. The first pending value is a value greater than the initial allocated storage space size for the first computing task; that is, the first pending value is greater than the third value. Since the first computing task fails on its first execution due to insufficient storage space in the third resource container, to reduce the probability of the first computing task failing again, the storage space size of the resource containers used to execute the first computing task subsequently must be greater than the third value. Therefore, the first pending value can be used as an alternative value to determine the lower limit of the second value. Based on the third value, the compensation value for the storage space corresponding to the initial execution failure can be determined according to the number of retries, thus obtaining a first pending value larger than the third value. For example, the first pending value can be determined by the following formula:

[0159] mem2=startMem+const2×(tryNum+1)(Formula 11)

[0160] Wherein, mem2 is the first undetermined value, startMem is the third value, const2 is the second fixed parameter, which can be set by those skilled in the art according to the task calculation type, and tryNum is the number of times the first calculation task is re-executed.

[0161] Based on the task completion parameters and the amount of input data for the first computation task, a second undetermined value is determined. This second undetermined value serves as a candidate for the second value. This process can be referenced from the steps described above for determining the second value, and will not be repeated here. Then, the maximum value between the first and second undetermined values ​​is selected as the second value suitable for the storage space of the first computation task.

[0162] Therefore, to further reduce the failure probability of executing the first computing task, a first pending value is determined based on the third value of the storage space corresponding to the first execution failure and the number of retries. The value determined by the task completion parameters and the amount of input data is used as the second pending value. The first pending value determines the lower limit of the second value. In the actual storage space allocation process, the larger of the two pending values ​​is selected as the second value, so that the storage space allocated to the first computing task in the future is greater than the storage space of the first execution failure. This reduces the possibility that the first computing task may fail again because the second pending value may be smaller than the third value. Especially when the number of retries is small, the second pending value is small, and the first pending value can play a supporting role for the second pending value, thereby improving the success rate of the subsequent execution of the first computing task and thus improving the resource utilization efficiency of the computing node.

[0163] An example of a storage space allocation method:

[0164] For ease of understanding, the following example illustrates the storage space allocation method mentioned above by taking the example of a Task failing due to insufficient memory space. This Task is the first computational task.

[0165] See Figure 6When the management node determines that a Task failed due to insufficient memory (OOM) by storing error messages, it filters out multiple successful Tasks of the same type that have already been completed based on the Task's task computation type. Then, it calculates the average memory usage and average input data volume for each successful Task. This average memory usage can be represented by the average storage space usage. Specifically, it first obtains the task completion parameters for each successful Task, and then calculates the average storage space usage based on the historical storage space usage values ​​identified by these parameters. Simultaneously, the management node also obtains the number of retries and the amount of input data for each Task. It processes the average storage space usage, average input data volume, number of retries, and the Task's input data volume using a memory estimation algorithm, and then requests a new resource container based on the memory size calculated by the algorithm. The Task is then allocated to the new resource container and re-executed.

[0166] Execution logic of memory estimation algorithm:

[0167] Specifically, the execution logic of this memory estimation algorithm (MemEstimate) is as follows:

[0168] Algorithm: MemEstimate(avgDataInput:Int,avgMemUsage:Int,tryNum:Int,taskDataInput:Int,startMem:Int)

[0169] 1:alpha←avgMemUsage / (avgDataInput^2)

[0170] 2:beta←avgMemUsage / avgDataInput

[0171] 3: margin ← const 1 * (tryNum^2)

[0172] 4:mem1←alpha*(taskDataInput^2)+beta*taskDataInput+margin

[0173] 5:mem2←startMem+const2*(tryNum+1)

[0174] 6: return max(mem1, mem2)

[0175] The inputs to this memory estimation algorithm include: the average input data size of the target computation task set (avgDataInput), the average storage space usage of the target computation task set (avgMemUsage), the number of retries for the Task (tryNum), the input data size of the Task (taskDataInput), and the memory space size of the resource container when the Task fails to execute for the first time (startMem). All of these inputs are of type Int.

[0176] The proportional parameter alpha is determined in step 1. alpha is used to indicate the exponential nonlinear mean relationship between the target computing task set and the data volume and storage space. Thus, the data expansion characteristics of completed tasks of the same type can be transferred to the memory space estimation process of the Task through alpha.

[0177] Step 2 determines the proportional parameter beta, which indicates the linear mean relationship between the target computation task set and the data volume and storage space. Thus, beta can be used to migrate the linear storage relationship of completed tasks of the same type to the memory space estimation process of the Task.

[0178] Step 3 determines the retry parameter weight margin, where const1 is the first fixed parameter. The more times the Task is retried, the larger the margin becomes.

[0179] Step 4, based on alpha, beta, and margin, combined with the task's taskDataInput, determines an alternative memory value to allocate for the task.

[0180] Step 5 determines another alternative memory value to allocate for the Task based on tryNum and startMem.

[0181] Step 6 returns the largest of the two candidate memory values ​​as the memory space size for allocating the Task, where const2 is the second fixed parameter.

[0182] Therefore, by using the average input data volume and average storage space usage, the exponential nonlinear mean and linear mean relationships determined by completed tasks of the same type are applied to the memory estimation process of Task. At the same time, by determining the weight of the number of retries and adjusting memory, the possibility of Task failing again is reduced. Furthermore, based on the memory space size of the resource container when the first execution fails, another alternative memory size is determined, further reducing the possibility of Task failing again.

[0183] Example of automated task error message identification:

[0184] As mentioned above, when the first computing task fails due to insufficient storage space, the first computing node sends a task error message to the management node. In related technologies, the cause of this task error message usually needs to be manually investigated to determine whether the first computing task failed due to insufficient storage space, thus identifying the task error message as a storage error message. This investigation process is manual and inefficient. Therefore, this application provides an automated method for identifying task error messages. The method involves obtaining the task error message sent by the first computing node for the first computing task. A task error message is a message reporting a computing task execution failure. A storage error message is a type of task error message used to indicate that the first computing task failed due to insufficient storage space.

[0185] When the first computing node detects that the first computing task has failed, it will determine the reason for the failure and then encapsulate it in the task error message with an identifier. This identifier is used to indicate the reason for the failure of the first computing task.

[0186] In this embodiment of the application, after receiving a task error message, the management node can parse the content of the task error message. For example, it can first extract the identifier of the error stack information included in the task error message, and then perform string comparison or semantic analysis on the identifier to determine the reason for the failure of the first computing task.

[0187] In response to a specified identifier in the task error message indicating that execution failed due to insufficient storage space, the task error message is determined to be a storage error message, and then subsequent steps for allocating storage space are executed. This specified identifier is one of several identifiers used to indicate that the reason for the failure of the first computation task is insufficient storage space.

[0188] It should be noted that this designated identifier can not only carry semantic information about insufficient storage space, but also refine the semantic information based on different types of storage space, thereby more accurately describing the reason for task execution failure. For example, this designated identifier can be used to indicate that the first computing task failed due to insufficient memory space, or it can be used to indicate that the first computing task failed due to insufficient buffer space. Thus, the management node can perform targeted storage space allocation based on the refined designated identifier, reducing the probability of the first computing task failing again.

[0189] An example of automated task error message identification:

[0190] The following example illustrates the processing procedures for computing nodes and management nodes, using the case of insufficient memory space causing computing task execution failure.

[0191] See Figure 7This figure is a schematic diagram of a computing node reporting message according to an embodiment of this application. In response to a computing task execution failure, the computing node containing the task captures the error process of the failure and determines whether the failure is due to insufficient memory (OutOfMemory). If the failure is due to insufficient memory, a TaskOOM (Computing Task Memory Insufficient) identifier is generated and encapsulated in the task error message. If the failure is not due to insufficient memory, other types of identifiers are encapsulated in the task error message. Then, the task error message is serialized and reported to the management node.

[0192] See Figure 8 This figure is a schematic diagram of a management node parsing message according to an embodiment of this application. After receiving a task error message from a computing node, the management node performs deserialization parsing to determine whether it includes the specified identifier of TaskOOM. If so, the computing task is marked, indicating that the computing task failed due to insufficient memory, and the task error message is a stored error message. If the identifier is not included and is not TaskOOM, other processing methods are adopted.

[0193] Therefore, the management node identifies the identifier carried in the task error message by parsing the task error message from the first computing node. This identifier indicates the specific reason for the failure of the first computing task. If the task error message carries a specified identifier indicating insufficient storage space, the task error message can be classified as a storage error message, and the relevant steps for subsequent storage space allocation can be automatically executed. Compared with related technologies, the embodiments of this application skip the manual investigation process of task error messages, which not only avoids subjective misjudgment and improves the accuracy of identifying task error messages, but also simplifies the process from obtaining task error messages to taking corresponding processing measures, thereby improving the efficiency of storage space allocation.

[0194] Example of initial storage space allocation:

[0195] The above-mentioned storage space allocation methods are all based on the reallocation of storage space when the first computing task fails. This application embodiment also provides a specific implementation method for the initial allocation of storage space for the first computing task.

[0196] In response to determining the amount of input data for the first computing task, a target value for the storage space applicable to the first computing task is determined based on the task completion parameters and the amount of input data for the first computing task.

[0197] The target value is the storage space size of the resource container initially allocated for the first computing task. As can be seen from the aforementioned method for determining the second value, the amount of input data for the first computing task needs to be determined before estimating the required storage space. The target computing task is a completed computing task of the same type; therefore, the target computing task can be a computing task completed before the execution of the first computing task, and its task completion parameters can be obtained when initially allocating storage space for the first computing task. In other words, during the initial allocation of the resource container for the first computing task, as long as the amount of input data for the first computing task can be determined, the target value applicable to the first computing task can be determined using the task completion parameters and the amount of input data of the target computing task. The calculation process for this target value is detailed in the aforementioned process for determining the second value, and will not be repeated here.

[0198] After obtaining the target value, a third resource container with storage space equal to the target value is created on the third computing node, and the first computing task is assigned to the third resource container for initial execution. The third computing node is the computing node that executes the first computing task for the first time, and the third resource container is the resource container allocated when the first computing task is executed for the first time.

[0199] Therefore, given the amount of input data for the first computing task, the storage space for the first computing task can be allocated differently by determining the target value. Thus, when the first computing task is executed for the first time, it can be allocated to a third resource container that is more suitable for the actual storage space required. This reduces the probability of the first computing task failing from the very beginning, further improves resource utilization, and greatly improves the overall execution efficiency of the data processing task.

[0200] Example scenarios of storage space allocation methods:

[0201] See Figure 9 The figure is a flowchart illustrating an application scenario of a storage space allocation method provided in an embodiment of this application.

[0202] First, the memory space size of the resource container in each compute node is determined to be 2GB based on the initial memory parameters. Then, multiple resource containers with a memory space size of 2GB are created on each compute node according to the initial memory parameters.

[0203] If the first computation task fails due to insufficient memory, it will report a task error message with error stack information to the management node. After receiving the task error message, the management node will parse the error stack information and determine that the reason for the failure of the first computation task is insufficient memory. Then, the management node will obtain the input data volume of the first computation task and the task completion parameters of similar completed computation tasks. These task completion parameters are used to identify the input data volume and peak memory usage of similar completed computation tasks. The peak memory usage is a historical storage space usage value.

[0204] The input data volume and task completion parameters of the first computing task are processed by a memory estimation algorithm to obtain a larger new memory parameter. This memory parameter indicates that the memory space of the resource container for re-executing the first computing task is 4GB. Finally, a second resource container with a memory space of 4GB is created according to the new memory parameter, and the first computing task is allocated to the second resource container for re-execution. The processing procedure of the memory estimation algorithm can refer to the execution logic of the memory estimation algorithm in the aforementioned embodiment, and will not be repeated here.

[0205] Examples of related devices:

[0206] In response to the storage space allocation method described above, this application also provides a corresponding storage space allocation device so that the above storage space allocation method can be applied and implemented in practice.

[0207] See Figure 10 , Figure 10 This is a schematic diagram of a storage space allocation device provided in an embodiment of this application. Figure 10 As shown, the storage space allocation device 1000 includes: an acquisition unit 1001, a determination unit 1002, and an allocation unit 1003;

[0208] The acquisition unit 1001 is used to acquire a storage error message sent by the first computing node for the first computing task. The first computing node is configured with a first resource container for executing the first computing task. The storage space allocated to the first resource container is a first value. The first computing task is one of multiple computing tasks obtained by decomposing data processing tasks.

[0209] The acquisition unit 1001 is further configured to respond to determining through the storage error message that the first computing task failed to execute due to insufficient storage space in the first resource container, and to acquire the task completion parameters of the target computing task based on the task computing type of the first computing task. The target computing task is a completed computing task with the same task type as the first computing task. The task completion parameters are used to identify the historical input data volume and historical storage space usage value of the target computing task.

[0210] The determining unit 1002 is used to determine a second value of the storage space applicable to the first computing task based on the task completion parameters and the amount of input data of the first computing task, wherein the second value is greater than the first value.

[0211] The allocation unit 1003 is used to create a second resource container with storage space of the second value, and allocate the first computing task to the second resource container for re-execution.

[0212] As can be seen from the above technical solutions, the storage space allocation device provided in this application includes an acquisition unit, a determination unit, and an allocation unit. When processing a data processing task, the data processing task is decomposed into multiple computing tasks, which can be processed in parallel using distributed computing. The first computing task is configured to be executed in the first resource container of the first computing node. If the acquisition unit obtains a storage error message indicating that the first computing task failed due to insufficient storage space, the acquisition unit can obtain the task completion parameters of a target computing task with the same task computing type as the first computing task from the completed computing tasks. These task completion parameters identify the amount of historical input data and the historical storage space occupied when the target computing task was successfully completed. Since the target computation task is a completed computation task of the same type as the first computation task, its task completion parameters are of substantial guiding significance for the successful execution of the first computation task. Considering that the amount of input data involved in different computation tasks of the same type, i.e., the amount of data to be computed, will differ, the determining unit can estimate how much storage space is needed to successfully compute the input data under the standard of successful execution of the target computation task based on the task completion parameters and the amount of input data of the first computation task, and thus obtain a second value that is larger than the first value. The allocation unit creates a second resource container with the storage space of the second value, and allocates the first computation task to the second resource container for re-execution. Since the determined second value is closer to the actual storage space required for the successful execution of the first computation task, it not only greatly increases the probability of the first computation task being successfully executed and effectively improves resource utilization efficiency, but also, this differentiated storage space allocation method for different computation tasks means that when the first computation task fails as described above, only the storage space of the first computation task needs to be reallocated, without affecting the normal execution of other computation tasks in the data processing task, effectively reducing the impact of computation task execution failure on the overall latency of data processing task completion.

[0213] As one possible implementation, the determining unit 1002 includes a linear unit and a first determining unit:

[0214] The linear unit is used to determine the linear relationship between the target computing task and the storage space based on the historical input data volume and the historical storage space usage value identified by the task completion parameter.

[0215] The first determining unit is used to determine a second value of the storage space applicable to the first computing task based on the linear relationship and the amount of input data of the first computing task.

[0216] As one possible implementation, the first determining unit includes a nonlinear unit and a first sub-determining unit:

[0217] The nonlinear unit is used to determine the exponential nonlinear relationship between the data volume and storage space for the target computing task by using the historical input data volume and the historical storage space occupancy value.

[0218] The first sub-determining unit is used to determine a second value of the storage space applicable to the first computing task based on the linear relationship, the exponential nonlinear relationship, and the amount of input data of the first computing task.

[0219] As one possible implementation, the exponential nonlinear relationship is a second-order exponential nonlinear relationship.

[0220] As one possible implementation, the acquisition unit 1001 is specifically used for:

[0221] Based on the task calculation type of the first calculation task, the task completion parameters of each historical calculation task in the target calculation task set are obtained. The historical calculation tasks are completed calculation tasks with the same task type as the first calculation task.

[0222] The linear unit is specifically used for:

[0223] By using the historical input data volume and historical storage space usage values ​​identified by the task completion parameters of the historical computing tasks, the linear mean relationship between the data volume and storage space for the target computing task set is determined.

[0224] As one possible implementation, the device further includes a weight determination unit:

[0225] The weight determination unit is used to determine the retry count weight based on the number of times the first computing task is re-executed;

[0226] The first determining unit is specifically used for:

[0227] Based on the linear relationship, the retry count weight, and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined.

[0228] As one possible implementation, the first computation task is executed for the first time through a third resource container and fails due to insufficient storage space. The third resource container is allocated storage space of a third value. The device further includes a second determining unit.

[0229] Based on the third value and the number of times the first calculation task was re-executed, a first undetermined value is determined;

[0230] The determining unit 1002 includes a third determining unit and a fourth determining unit:

[0231] The third determining unit is used to determine the second undetermined value by using the task completion parameters and the amount of input data of the first calculation task;

[0232] The fourth determining unit is used to determine the second value of the storage space applicable to the first computing task from the maximum value of the first undetermined value and the second undetermined value.

[0233] As one possible implementation, the acquisition unit 1001 includes a message acquisition unit, a parsing unit, and a message determination unit:

[0234] The message acquisition unit is used to acquire the task error message sent by the first computing node for the first computing task;

[0235] The parsing unit is used to parse the content of the task error message;

[0236] The message determination unit is configured to determine the task error message as the storage error message in response to a specified identifier in the task error message indicating that the execution failed due to insufficient storage space.

[0237] As one possible implementation, when initially allocating the first computing task, the apparatus further includes a target value determination unit and an initial allocation unit:

[0238] The target value determination unit is used to determine a target value of the storage space applicable to the first computing task in response to determining the amount of input data of the first computing task, based on the task completion parameters and the amount of input data of the first computing task.

[0239] The initial allocation unit is used to request the creation of a third resource container with storage space of the target value on the third computing node, and to allocate the first computing task to the third resource container for initial execution.

[0240] As one possible implementation, the storage space identified by the storage error message is at least one of the memory space or buffer space allocated to the first resource container.

[0241] As one possible implementation, the allocation unit 1003 is specifically used for:

[0242] On the first computing node, request the creation of a second resource container with a storage space equal to the second value; or

[0243] A second resource container with a storage space of the second value is requested to be created on a second computing node that is different from the first computing node.

[0244] This application also provides a computer device, including a terminal device or a server, in which the aforementioned storage space allocation device can be configured. The computer device will now be described in conjunction with the accompanying drawings.

[0245] If the computer device is a terminal device, please refer to Figure 11 As shown, this application provides a terminal device, taking a mobile phone as an example:

[0246] Figure 11 The diagram shown is a block diagram of a portion of the structure of a mobile phone provided in an embodiment of this application. (Reference) Figure 11 The mobile phone includes components such as a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a Wi-Fi module 1470, a processor 1480, and a power supply 1490. Those skilled in the art will understand that... Figure 11 The mobile phone structure shown does not constitute a limitation on the mobile phone and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0247] The following is combined with Figure 11 A detailed introduction to each component of a mobile phone:

[0248] The RF circuit 1410 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1480; in addition, it transmits uplink data to the base station.

[0249] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various mobile phone functions and data processing by running the software programs and modules stored in the memory 1420. The memory 1420 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0250] The input unit 1430 can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432.

[0251] The display unit 1440 can be used to display information input by the user or information provided to the user, as well as various menus of the mobile phone. The display unit 1440 may include a display panel 1441.

[0252] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors.

[0253] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the user and the mobile phone.

[0254] WiFi is a short-range wireless transmission technology. Through the WiFi module 1470, mobile phones can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access.

[0255] The processor 1480 is the control center of the mobile phone. It connects to various parts of the mobile phone through various interfaces and lines. It performs various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420 and calling data stored in the memory 1420.

[0256] The phone also includes a power supply 1490 (such as a battery) that powers the various components.

[0257] In this embodiment, the processor 1480 included in the terminal device is also used to execute the steps in the methods of the various embodiments of this application.

[0258] If the computer device is a server, this application embodiment also provides a server; please refer to [link to relevant documentation]. Figure 12 As shown, Figure 12This is a structural diagram of a server 1500 provided in an embodiment of this application. The server 1500 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1522 (e.g., one or more processors) and a memory 1532, and one or more storage media 1530 (e.g., one or more mass storage devices) for storing application programs 1542 or data 1544. The memory 1532 and storage media 1530 can be temporary or persistent storage. The program stored in the storage media 1530 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1522 may be configured to communicate with the storage media 1530 and execute the series of instruction operations in the storage media 1530 on the server 1500.

[0259] Server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input / output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0260] The steps performed by the server in the above embodiments can be based on Figure 12 The server structure shown.

[0261] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0262] This application also provides a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the method provided in the above embodiments.

[0263] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0264] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0265] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0266] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Moreover, based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A storage space allocation method, characterized in that, The method includes: Obtain a storage error message sent by the first computing node for the first computing task. The first computing node is configured with a first resource container for executing the first computing task. The storage space allocated to the first resource container is a first value. The first computing task is one of multiple computing tasks obtained by decomposing data processing tasks. In response to determining through the storage error message that the first computing task failed to execute due to insufficient storage space in the first resource container, based on the task computing type of the first computing task, the task completion parameters of the target computing task are obtained. The target computing task is a completed computing task with the same task type as the first computing task. The task completion parameters are used to identify the historical input data volume and historical storage space usage value of the target computing task. Based on the task completion parameters and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined, wherein the second value is greater than the first value. Create a second resource container with storage space equal to the second value, and assign the first computing task to the second resource container for re-execution.

2. The method according to claim 1, characterized in that, The step of determining a second value for the storage space applicable to the first computing task based on the task completion parameters and the amount of input data for the first computing task includes: By using the historical input data volume and historical storage space usage value identified by the task completion parameters, the linear relationship between the data volume and storage space for the target computing task is determined. Based on the linear relationship and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined.

3. The method according to claim 2, characterized in that, The step of determining a second value for the storage space applicable to the first computing task based on the linear relationship and the amount of input data for the first computing task includes: By using the historical input data volume and the historical storage space usage value, the exponential nonlinear relationship between the data volume and storage space for the target computing task is determined. Based on the linear relationship, the exponential nonlinear relationship, and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined.

4. The method according to claim 3, characterized in that, The exponential nonlinear relationship is a second-order exponential nonlinear relationship.

5. The method according to claim 2, characterized in that, The step of obtaining the task completion parameters of the target computing task based on the task computing type of the first computing task includes: Based on the task calculation type of the first calculation task, the task completion parameters of each historical calculation task in the target calculation task set are obtained. The historical calculation tasks are completed calculation tasks with the same task type as the first calculation task. The determination of the linear relationship between data volume and storage space usage for the target computation task, based on the historical input data volume and historical storage space usage values ​​identified by the task completion parameters, includes: By using the historical input data volume and historical storage space usage values ​​identified by the task completion parameters of the historical computing tasks, the linear mean relationship between the data volume and storage space for the target computing task set is determined.

6. The method according to claim 2, characterized in that, The method further includes: The retry count weight is determined based on the number of times the first computation task is re-executed; The step of determining a second value for the storage space applicable to the first computing task based on the linear relationship and the amount of input data for the first computing task includes: Based on the linear relationship, the retry count weight, and the amount of input data for the first computing task, a second value for the storage space applicable to the first computing task is determined.

7. The method according to claim 1, characterized in that, The first computation task is executed for the first time through a third resource container and fails due to insufficient storage space. The third resource container is allocated storage space of a third numerical value. The method further includes: Based on the third value and the number of times the first calculation task was re-executed, a first undetermined value is determined; The step of determining a second value for the storage space applicable to the first computing task based on the task completion parameters and the amount of input data for the first computing task includes: The second undetermined value is determined by using the task completion parameters and the amount of input data for the first calculation task; The second value is determined from the maximum value between the first undetermined value and the second undetermined value as the storage space applicable to the first computing task.

8. The method according to any one of claims 1-7, characterized in that, The step of obtaining the stored error message sent by the first computing node for the first computing task includes: Obtain the task error message sent by the first computing node for the first computing task; The task error message is parsed. In response to the task error message carrying a specified identifier indicating that the execution failed due to insufficient storage space, the task error message is determined to be the storage error message.

9. The method according to any one of claims 1-7, characterized in that, When initially assigning the first computing task, the method further includes: In response to determining the amount of input data for the first computing task, a target value for the storage space applicable to the first computing task is determined based on the task completion parameters and the amount of input data for the first computing task. On the third computing node, a third resource container with storage space of the target value is created, and the first computing task is assigned to the third resource container for initial execution.

10. The method according to any one of claims 1-7, characterized in that, The storage space identified by the storage error message is at least one of the memory space or buffer space allocated to the first resource container.

11. The method according to any one of claims 1-7, characterized in that, The creation of a second resource container with storage space of the second value includes: On the first computing node, request the creation of a second resource container with a storage space equal to the second value; or A second resource container with a storage space of the second value is requested to be created on a second computing node that is different from the first computing node.

12. A storage space allocation device, characterized in that, The device includes: an acquisition unit, a determination unit, and an allocation unit; The acquisition unit is used to acquire a storage error message sent by the first computing node for the first computing task. The first computing node is configured with a first resource container for executing the first computing task. The storage space allocated to the first resource container is a first value. The first computing task is one of multiple computing tasks obtained by decomposing data processing tasks. The acquisition unit is further configured to respond to determining through the storage error message that the first computing task failed to execute due to insufficient storage space in the first resource container, and based on the task computing type of the first computing task, to acquire the task completion parameters of the target computing task, wherein the target computing task is a completed computing task with the same task type as the first computing task, and the task completion parameters are used to identify the historical input data volume and historical storage space usage value of the target computing task. The determining unit is used to determine a second value of the storage space applicable to the first computing task based on the task completion parameters and the amount of input data of the first computing task, wherein the second value is greater than the first value. The allocation unit is used to create a second resource container with storage space of the second value, and to allocate the first computing task to the second resource container for re-execution.

13. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs; The processor is configured to perform the method according to any one of claims 1-11 according to the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that, when executed by a computer device, performs the method described in any one of claims 1-9.

15. A computer program product comprising a computer program, which, when run on a computer device, causes the computer device to perform the method of any one of claims 1-11.

Images