Method and Apparatus for Automatic Recovery of Tasks Using Execution Failure-Based Resource Requirement Adjustment

Abstract

The present invention relates to a method and apparatus for automatic job recovery using resource requirement adjustment based on execution failure. The present invention comprises the steps of: detecting when an execution failure event caused by resource shortage occurs during job execution; quantitatively calculating the amount of resource shortage by analyzing the execution log of the job in response to the execution failure event; converting the calculated amount of resource shortage into a logical resource unit independent of the type of physical acceleration device or memory capacity; adjusting the resource requirement for the job based on the logical resource unit; generating a new execution specification corresponding to the job by reflecting the adjusted resource requirement; and re-executing the job according to the new execution specification. By utilizing execution failures caused by resource shortage occurring during job execution in a distributed computing environment not as simple errors but as meaningful feedback signals obtainable from the execution results, the resource requirement of the job can be automatically adjusted to recover the job.

Description

Technology Field

[0001] The present invention relates to the field of task execution control and resource management technology in a distributed computing environment, and more specifically, to a method and apparatus for detecting execution failures caused by resource shortages occurring in container execution environments, artificial intelligence / machine learning training and inference pipelines, HPC (High Performance Computing) task scheduling environments, etc., quantitatively calculating the amount of resource shortage from failure logs, and performing automatic recovery by automatically adjusting the resource requirements of the task execution specification based on this and re-executing the task. Background Technology

[0003] With the recent proliferation of training and inference of large-scale deep learning models, large-scale data processing, and GPU-based parallel computing, job or pipeline execution in distributed environments is becoming commonplace. These jobs are executed on a container basis, and before execution, an execution specification including resource requirements (CPU, memory, GPU, etc.) is submitted, and the control plane and scheduler place and execute the jobs according to the specification.

[0004] However, actual memory usage for tasks based on AI / ML frameworks (such as PyTorch and TensorFlow) can vary due to various factors during execution, including input data, batch size, model structure, and cache / buffers. In particular, problems such as abnormal job termination caused by errors resulting from insufficient GPU memory are frequent. Traditionally, such Out-Of-Memory (OOM) failures were treated as simple "execution errors," leading users to respond by manually modifying resource requests and resubmitting, or by conservatively requesting excessive resources. Consequently, this results in reduced productivity due to repeated failures, wasted resources, compromised fairness in multi-tenant environments, and increased costs.

[0005] Meanwhile, orchestration systems like Kubernetes often intentionally do not provide or restrict automatic recovery that arbitrarily changes the resource requirements of running containers, due to their design philosophy. Runtime resource changes can compromise the immutability of Pod Specs, thereby reducing the reliability of scheduler decisions; they can also invalidate scheduler determinism, leading to unpredictable outcomes such as node over-allocation; and in multi-tenant environments, they can be interpreted as specific tasks automatically acquiring priority incrementally, potentially sparking fairness controversies. Furthermore, methods that modify internal scheduler logic or directly edit the runtime carry a high degree of system intrusion, resulting in a significant operational burden. Therefore, there is a need for technology capable of automatically recovering from recurring resource shortage failures and refining resource planning while respecting the execution contracts of existing systems.

[0006] Prior art patent is Korean Registered Patent No. 10-2790978 (Flexible GPU resource scheduling method in a large-scale container operation environment), but this prior art patent merely discloses a scheduling technology for efficiently allocating GPU resources in advance in a large-scale container environment, and does not disclose an execution failure-based automatic recovery technology that analyzes execution failures caused by resource shortages, quantitatively calculates the amount of insufficient resources, and automatically adjusts and regenerates the execution specification itself to recover the task. The problem to be solved

[0008] The present invention is proposed to solve the problems of the prior art described above, and aims to provide a technology that automatically adjusts the resource requirements of a task and recovers execution by utilizing execution failures caused by resource shortages in a distributed computing environment not as simple errors, but as meaningful feedback signals obtainable from the execution results.

[0009] Specifically, the present invention aims to quantitatively calculate the amount of resource shortage by analyzing execution logs in response to a resource shortage execution failure that occurs during task execution, and to convert the calculated amount of resource shortage into a logical resource unit independent of the physical resource environment and reflect it in the task's resource requirements, thereby enabling the task to be automatically re-executed according to the modified execution specifications.

[0010] In addition, the present invention aims to reduce repetitive execution failures and improve resource efficiency while maintaining the traceability and reproducibility of execution history by performing automatic recovery in a manner that updates execution specifications between executions without changing the runtime state of the task currently being executed. means of solving the problem

[0012] The method for automatic recovery of a task using resource requirement adjustment based on execution failure according to the present invention comprises: a step of detecting when an execution failure event caused by resource shortage occurs during task execution; a step of quantitatively calculating the amount of resource shortage by analyzing the execution log of the task in response to the execution failure event; a step of converting the calculated amount of resource shortage into a logical resource unit independent of the type of physical acceleration device or memory capacity; a step of adjusting the resource requirement for the task based on the logical resource unit; a step of generating a new execution specification corresponding to the task by reflecting the adjusted resource requirement; and a step of re-executing the task according to the new execution specification.

[0013] The automatic job recovery device using the execution failure-based resource requirement adjustment of the present invention includes a memory that stores one or more program instructions and one or more processors that execute the program instructions stored in the memory. The processors are controlled to detect an execution failure event caused by a resource shortage during job execution, and are controlled to quantitatively calculate the resource shortage amount by analyzing the execution log of the job in response to the execution failure event. They are controlled to convert the calculated resource shortage amount into a logical resource unit independent of the type of physical acceleration device or memory capacity. They are controlled to adjust the resource requirement amount of the job based on the logical resource unit. They are controlled to generate a new execution specification reflecting the adjusted resource requirement amount, wherein the new execution specification includes execution history identification information and update information. After verifying a resource allocation policy or a resource quota limit, the job is re-executed according to the new execution specification only when the policy or resource quota limit is satisfied. Effects of the invention

[0015] According to the present invention, execution failures caused by resource shortages occurring during task execution in a distributed computing environment are not treated as simple errors but are utilized as meaningful feedback signals obtainable from the execution results, thereby allowing the task to be recovered by automatically adjusting the resource requirements of the task. Accordingly, repetitive execution failures can be effectively reduced.

[0016] Furthermore, the present invention quantitatively calculates the resource shortage from execution logs and converts the calculated shortage into a logical resource unit independent of the physical resource environment, reflecting this in the adjustment of resource requirements. This enables precise resource allocation tailored to the characteristics of the task. As a result, it is possible to prevent over-allocation of resources beyond what is necessary and improve resource utilization efficiency.

[0017] Furthermore, the present invention performs automatic recovery by updating execution specifications between executions without altering the runtime state of the running task or the container execution environment, thereby maintaining the stability of execution contracts and scheduling decisions in a distributed computing system. Accordingly, it has low system penetration and is applicable to various distributed environments.

[0018] Furthermore, by including execution history identification information and update information in the execution specification and managing them, the present invention ensures the traceability and reproducibility of the work execution history and facilitates repeated execution or post-mortem analysis of the same task.

[0019] Consequently, the present invention provides the effect of improving the stability of task execution in a distributed computing environment, minimizing manual intervention by operators, and simultaneously improving resource efficiency and operational efficiency. Brief explanation of the drawing

[0020] FIG. 1 is a diagram illustrating the configuration of a distributed computing system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating the internal functional configuration of an automatic work recovery device according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the overall flowchart of an automatic work recovery method according to one embodiment of the present invention. FIG. 4 is a detailed flowchart illustrating the step of calculating the resource shortage amount according to one embodiment of the present invention. FIG. 5 is a flowchart illustrating the verification step of the resource requirement adjustment condition according to one embodiment of the present invention. Specific details for implementing the invention

[0021] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0022] Embodiments according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0023] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described herein, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached to this specification.

[0026] FIG. 1 is a diagram illustrating the configuration of a distributed computing system according to an embodiment of the present invention.

[0027] Referring to FIG. 1, a distributed computing system (10) according to one embodiment of the present invention is an environment for executing and managing tasks using a plurality of computing resources, and may include a container (200) for executing tasks, a scheduler (300) for performing task scheduling and resource allocation, a task execution node (400) where task execution takes place, and an execution log collection unit (500) for collecting and storing execution logs generated during the task execution process. Within the container (200), a machine learning framework or a computation framework for performing machine learning tasks or computation tasks may be executed, and the framework may output an error message or execution log caused by a lack of resources during task execution.

[0028] A distributed computing system (10) includes a task automatic recovery device (100) configured to automatically recover a task by adjusting resource requirements in response to execution failures occurring during task execution. The task automatic recovery device (100) may be implemented as a central server, a control node, or a management agent as a component of the distributed computing system (10), and may be placed in various physical or logical locations according to embodiments of the present invention.

[0029] The task automatic recovery device (100) is composed of a memory (110), a processor (120), a communication module (130), and an input / output interface (140). The task automatic recovery device (100) may be configured to communicate information and / or data through a network using the communication module (130).

[0030] The memory (110) may include any non-transient computer-readable recording medium. According to one embodiment, the memory (110) may include random access memory (RAM), read-only memory (ROM), a disk drive, a solid state drive (SSD),

[0031] It may include a permanent mass storage device such as flash memory. As another example, a permanent mass storage device such as ROM, SSD, flash memory, or disk drive may be included in the task automatic recovery device (100) as a separate permanent storage device distinct from the memory (110). Additionally, the memory (110) may store an operating system and at least one program code (e.g., code for structured information extraction using a deep learning model installed and running in the task automatic recovery device (100), code for training a deep learning model, etc.).

[0032] These software components may be loaded from a computer-readable recording medium separate from the memory (110). This separate computer-readable recording medium may include a recording medium that can be directly connected to the task automatic recovery device (100), for example, a computer-readable recording medium such as a floppy drive, disk, tape, DVD / CD-ROM drive, memory card, etc. As another example, the software components may be loaded into the memory (110) via a communication module (130) rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory (110) based on a computer program (e.g., a program for extracting structured information using a deep learning model, training a deep learning model, etc.) that is installed by files provided through the communication module (130) by developers or a file distribution system that distributes installation files for the application.

[0033] A processor (120) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to a user terminal (not shown) or another external system by memory (110) or a communication module (130). For example, the one or more processors may be controlled to detect an execution failure event caused by a shortage of resources during task execution, to quantitatively calculate the amount of resource shortage by analyzing the execution log of the task in response to the execution failure event, to convert the calculated amount of resource shortage into a logical resource unit independent of the type of physical acceleration device or memory capacity, to control the resource requirement of the task to be adjusted based on the logical resource unit, to control the creation of a new execution specification reflecting the adjusted resource requirement, wherein the new execution specification includes execution history identification information and update information, and after verifying a resource allocation policy or resource quota limit, to control the task to be re-executed according to the new execution specification only if the policy or resource quota limit is satisfied.

[0034] The communication module (130) may provide a configuration or function for a user terminal (not shown) and an automatic task recovery device (100) to communicate with each other via a network, and may provide a configuration or function for the automatic task recovery device (100) to communicate with an external system (e.g., a separate cloud system). For example, control signals, commands, data, etc. provided under the control of the processor (120) of the automatic task recovery device (100) may be transmitted to the user terminal and / or the external system through the communication module (130) and the network, and through the communication module of the user terminal and / or the external system.

[0035] Additionally, the input / output interface (140) of the automatic task recovery device (100) may be a means for interfacing with a device (not shown) for input or output that is connected to the automatic task recovery device (100) or that the automatic task recovery device (100) may include. In FIG. 1, the input / output interface (140) is shown as an element configured separately from the processor (120), but is not limited thereto, and the input / output interface (140) may be configured to be included in the processor (120). The automatic task recovery device (100) may include more components than those of FIG. 1. However, there is no need to clearly illustrate most of the conventional technical components.

[0036] The processor (120) of the task automatic recovery device (100) may be configured to manage, process, and / or store information and / or data received from a plurality of user terminals and / or a plurality of external systems.

[0038] FIG. 2 is a diagram illustrating an internal functional configuration block diagram of an automatic work recovery device (100) according to an embodiment of the present invention.

[0039] Referring to FIG. 2, the processor (120) may include an execution failure detection unit (121), a resource shortage calculation unit (122), a resource unit conversion unit (123), a resource requirement adjustment unit (124), an execution specification generation unit (125), and a task re-execution control unit (126). Each of the above components may be a physically separated hardware component, or a software module functionally implemented by an instruction stored in memory (110) being executed by the processor (120).

[0040] The execution failure detection unit (121) is configured to detect whether an execution failure event caused by resource shortage has occurred during task execution. For example, the execution failure detection unit (121) can determine whether an execution has failed by receiving an error code, an error message, or exit status information transmitted from a container (200) or a machine learning framework (210).

[0041] The resource shortage calculation unit (122) is configured to quantitatively calculate the resource shortage by analyzing the execution log of the corresponding task when an execution failure event is detected. In one embodiment, the resource shortage calculation unit (122) can quantify the memory shortage directly caused by the execution failure by parsing the requested memory capacity, actual allocated memory capacity, or available memory information included in the error message.

[0042] The resource unit conversion unit (123) is configured to convert the calculated resource shortage amount into a logical resource unit independent of the type of physical acceleration device or memory capacity. The logical resource unit may include a fractional GPU unit, a virtual accelerator unit, or an abstract resource unit equivalent thereto, thereby enabling adjustment of the resource requirement amount that is not dependent on a specific hardware environment. In this specification, 'logical resource unit' refers to an abstract resource representation unit for expressing the resource requirement amount regardless of the type and capacity of the physical acceleration device.

[0043] The resource requirement adjustment unit (124) is configured to adjust the resource requirement for a task based on the resource shortage amount converted into a logical resource unit. At this time, the resource requirement adjustment unit (124) can determine whether to adjust the resource requirement and the scope of adjustment by considering resource allocation policies or resource quota limits predefined at the user, project, or system level.

[0044] The execution specification generation unit (125) is configured to generate a new execution specification corresponding to the task by reflecting the adjusted resource requirements. At this time, the execution specification generation unit (125) can generate a new execution specification that is distinct from the execution specification used in the previous execution without changing the runtime state or container execution environment of the task currently being executed. Additionally, the new execution specification may include execution history identification information to identify the previous execution specification and update information indicating that it has been updated by adjusting the resource requirements. For example, the execution specification may include a Job specification, a Pipeline specification, or similar task definition information, and may be expressed in YAML, JSON, or a similar format, but is not limited thereto.

[0045] The task re-execution control unit (126) is configured to control the re-execution of a task according to a newly generated execution specification. In one embodiment, the task re-execution control unit (126) may control the placement of a task reflecting the adjusted resource requirements in conjunction with the scheduler (300) to an appropriate task execution node (400).

[0046] In this way, the processor (120) according to the embodiment of the present invention can recover the task by automatically adjusting the resource requirements of the task through each component shown in FIG. 2 by utilizing the execution failure caused by resource shortage occurring during task execution in a distributed computing environment as a meaningful feedback signal.

[0048] FIG. 3 is a diagram illustrating the overall flowchart of an automatic work recovery method according to one embodiment of the present invention.

[0049] Referring to FIG. 3, the job automatic recovery device (100) detects when an execution failure event occurs due to a lack of resources during job execution (S301). In one embodiment, the execution failure event may be identified by the termination status of the container (200), an event of the orchestration system, or an Out-Of-Memory (OOM) related error message output from the machine learning framework / computation framework (210), and the execution failure event may include a job identifier, execution time, execution node identifier, container identifier, error code / string, etc.

[0050] When the above execution failure event is detected, the task automatic recovery device (100) collects and analyzes the execution log of the task in response to the execution failure event to quantitatively calculate the amount of resource shortage (S303). In one embodiment, the task automatic recovery device (100) may acquire logs through the execution log collection unit (500) or the log interface of the orchestration system, and may calculate the amount of shortage as a numerical value (e.g., MiB, GiB) by parsing information such as “requested memory capacity,” “available memory capacity,” and “cause of allocation failure” included in the error message. Additionally, if multiple error information is included in the log, information related to resource shortage that is the direct cause of the execution failure (e.g., GPU memory shortage) may be selected and calculated first.

[0051] Next, the task automatic recovery device (100) converts the calculated resource shortage into a logical resource unit independent of the type of physical acceleration device or memory capacity (S305). In one embodiment, the logical resource unit may be an abstraction unit defined for resource requirement adjustment, such as a fractional GPU unit or a virtual accelerator unit, and the task automatic recovery device (100) may convert the resource shortage into a logical resource unit based on predefined resource mapping information (e.g., conversion table, conversion factor, profile information). If necessary, the conversion result may be determined by reflecting a safety margin in the calculated shortage or conversion result to increase re-execution stability.

[0052] Subsequently, the task automatic recovery device (100) adjusts the resource requirements for the task based on the logical resource unit (S307). In one embodiment, the task automatic recovery device (100) calculates the incremental resource requirements by combining the resource requirements (e.g., GPU, memory, other acceleration resources) listed in the existing execution specification with the shortage amount converted into a logical resource unit, and determines the adjusted resource requirements by reflecting this. Additionally, if there is a resource allocation policy or resource quota limit defined in advance at the user / project / system level, the resource requirements may be controlled to be adjusted only within the range where adjustment is allowed by verifying the policy or quota limit, and if adjustment is not allowed, the device may be configured to withhold automatic recovery and provide a notification.

[0053] When the resource requirements are adjusted, the task automatic recovery device (100) generates a new execution specification corresponding to the task by reflecting the adjusted resource requirements (S309). In one embodiment, the new execution specification is generated to be distinct from the execution specification used in the previous execution, and may be generated by updating the execution specification between executions without directly changing the runtime state of the task currently running or the container execution environment. Additionally, the new execution specification may include execution history identification information to identify the previous execution specification, and update information indicating that it has been updated due to the adjustment of resource requirements. Here, the update information may be configured to include the cause of the update (e.g., OOM), the time of the update, resource requirements before and after the update, the number of updates, the reference value of the previous execution specification, etc.

[0054] Finally, the task automatic recovery device (100) re-executes the task according to the new execution specification (S311). In one embodiment, the task automatic recovery device (100) may control the relocation of the task by submitting the new execution specification through the submission interface of the scheduler (300) or the orchestration system, and may schedule the task to a node among the task execution nodes (400) that meets the adjusted resource requirements. Additionally, it may be configured to monitor the execution results after re-execution and repeat the process if additional execution failures occur.

[0056] FIG. 4 is a detailed flowchart illustrating the step of calculating the resource shortage amount according to one embodiment of the present invention.

[0057] Referring to FIG. 4, when a resource shortage calculation unit (122) of the task automatic recovery device (100) detects an execution failure event that occurred during task execution, it collects the execution log of the task (S401). The execution log may include error messages, warning messages, or status code information output from a container (200) or a machine learning framework (210).

[0058] The resource shortage calculation unit (122) parses the error message included in the execution log to identify the resource capacity requested during task execution and the resource capacity actually allocated or available resource information (S403). In one embodiment, the error message may include string information indicating the memory request amount, available memory capacity, or the cause of resource allocation failure.

[0059] The resource shortage calculation unit (122) calculates the resource shortage directly caused by the execution failure as a quantitative value based on the identified information (S405). For example, the memory shortage can be quantified by calculating the difference between the requested memory capacity and the actual allocated memory capacity.

[0060] In this way, the present invention quantitatively calculates the amount of resource shortage using information included in the execution log, thereby allowing the cause of execution failure to be determined abstractly, rather than being utilized as a concrete input value for adjusting the resource requirements of subsequent stages.

[0062] FIG. 5 is a flowchart illustrating the verification step of the resource requirement adjustment condition according to one embodiment of the present invention.

[0063] Referring to FIG. 5, the resource requirement adjustment unit (124) of the job automatic recovery device (100) compares the resource shortage amount converted into a logical resource unit with the policy or quota limit to verify whether resource requirement adjustment is allowed (S501, S503). If, as a result of verification, resource requirement adjustment is allowed, resource requirement adjustment is performed (S505), and if adjustment is not allowed, the job re-execution is suspended or a notification is provided to the administrator or user.

[0064] Specifically, the resource requirement adjustment unit (124) refers to resource allocation policies or resource quota limit information that are predefined at the user, project, or system level. The resource allocation policies or resource quota limit information may include the maximum resource capacity that can be allocated to a specific task or user, the incremental adjustment range, or whether automatic recovery is allowed. In this way, the present invention can prevent unlimited resource expansion and maintain the stability of resource management for the entire system by performing condition verification based on policies and quotas in the resource requirement adjustment step.

[0066] Meanwhile, according to one embodiment of the present disclosure, the various embodiments described above may be implemented as software comprising instructions stored on a machine-readable storage medium (e.g., a computer). The machine may include an electronic device according to the disclosed embodiments, which is a device capable of calling instructions stored from the storage medium and operating according to the called instructions. When instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or by using other components under the control of the processor. Instructions may include code generated or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, "non-transitory" means only that the storage medium does not contain a signal and is tangible, and does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

[0067] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0068] Additionally, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented as the processor itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein.

[0069] Meanwhile, computer instructions for performing processing operations of the device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. When computer instructions stored in such a non-transitory computer-readable medium are executed by the processor of a specific device, they cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and is readable by a device, rather than a medium that stores data for a short moment, such as a register, cache, or memory. Specific examples of a non-transitory computer-readable medium may include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, ROMs, etc.

[0070] Additionally, each component (e.g., module or program) according to the various embodiments described above may be composed of a single or multiple entities, and some of the aforementioned sub-components may be omitted, or other sub-components may be additionally included in the various embodiments. Generally or additionally, some components (e.g., module or program) may be integrated into a single entity to perform the same or similar functions as those performed by each of the respective components prior to integration. The operations performed by the module, program, or other components according to the various embodiments may be executed sequentially, in parallel, iteratively, or heuristically, or at least some operations may be executed in a different order, omitted, or other operations added.

[0071] The invention has been described with reference to embodiments illustrated in the drawings, but this is merely illustrative, and those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0073] 100 : Automatic task recovery device 110 : Memory 120 : Processor 121: Execution failure detection unit 122: Resource shortage calculation unit 123: Resource Unit Conversion Unit 124: Resource Requirement Adjustment Unit 125: Execution specification generation unit 126: Job re-execution control unit

Claims

Claim 1 A method for automatic job recovery using resource requirement adjustment based on execution failure, performed by at least one processor of an automatic job recovery device, comprising: detecting when an execution failure event caused by resource shortage occurs during job execution; quantitatively calculating the resource shortage amount by analyzing the execution log of the job in response to the execution failure event; converting the calculated resource shortage amount into a logical resource unit independent of the type of physical acceleration device or memory capacity; adjusting the resource requirement for the job based on the logical resource unit; and generating a new execution specification corresponding to the job by reflecting the adjusted resource requirement amount. A method for automatic task recovery using execution failure-based resource requirement adjustment, comprising the step of re-executing the task according to the new execution specification, wherein the step of generating the new execution specification is performed in such a manner that the task automatic recovery device generates a new execution specification distinct from the execution specification used in the previous execution without changing the runtime state or container execution environment of the task currently being executed, and wherein the new execution specification includes execution history identification information for identifying the previous execution specification and update information indicating that it has been updated by resource requirement adjustment from the previous execution specification, wherein the update information includes at least one of the cause of the update, the time of the update, the resource requirements before and after the update, the number of updates, and the reference value of the previous execution specification, wherein when determining the resource requirements, the incremental resource requirements are calculated by combining the resource requirements listed in the existing execution specification and the resource shortage amount converted into a logical resource unit, and are determined by reflecting the result, and wherein the resource shortage amount is a numerical value of the memory shortage amount caused by execution failure calculated based on the requested memory capacity and the actual allocated memory capacity or available memory information. Claim 2 A method for automatic task recovery using execution failure-based resource demand adjustment, wherein the step of quantitatively calculating the resource shortage amount in claim 1 is characterized by parsing memory request amount and available memory information included in an error message output from a machine learning framework or a computation framework to calculate the memory shortage amount as a quantitative value. Claim 3 A method for automatic job recovery using execution failure-based resource requirement adjustment according to claim 1, wherein the step of adjusting the resource requirement includes the step of verifying a resource allocation policy or resource quota limit predefined at the user, project, or system level based on the resource shortage amount converted into the logical resource unit by the automatic job recovery device, and is controlled so that the adjustment of the resource requirement is performed only when the resource allocation policy or resource quota limit is satisfied. Claim 4 delete Claim 5 A device for automatically recovering a task by adjusting resource requirements based on execution results in response to the execution failure of a task executed in a distributed computing environment, comprising: a memory storing one or more program instructions; and one or more processors executing the program instructions stored in the memory. The processor controls the detection of an execution failure event caused by a resource shortage during task execution, controls the quantitative calculation of the resource shortage amount by analyzing the execution log of the task in response to the execution failure event, controls the conversion of the calculated resource shortage amount into a logical resource unit independent of the type of physical acceleration device or memory capacity, controls the adjustment of the task's resource requirements based on the logical resource unit, controls the generation of a new execution specification reflecting the adjusted resource requirements, wherein the new execution specification includes execution history identification information and update information, controls the task to be re-executed according to the new execution specification only when the policy or resource quota limit is satisfied after verifying a resource allocation policy or resource quota limit, and when the processor generates the new execution specification, the task automatic recovery device does not change the runtime state or container execution environment of the task currently being executed, and the previous execution used It is performed by generating a new execution specification distinct from the execution specification, wherein the new execution specification includes execution history identification information for identifying the previous execution specification and update information indicating that it has been updated by adjusting the resource requirement from the previous execution specification, wherein the update information includes at least one of the cause of the update, the time of the update, the resource requirement before and after the update, the number of updates, and the reference value of the previous execution specification, and when determining the resource requirement, the incremental resource requirement is calculated by combining the resource requirement listed in the existing execution specification and the resource shortage amount converted into a logical resource unit, and is determined by reflecting this.An automatic job recovery device using execution failure-based resource demand adjustment, characterized in that the above resource shortage is a numerical value representing the amount of memory shortage caused by execution failure, calculated based on the requested memory capacity and the actual allocated memory capacity or available memory information.

Images