Distributed Computing Environment Update Scheduling System and Distributed Computing Environment Update Scheduling Method

Abstract

To update a distributed computing environment appropriately and efficiently during its operation. [Solution] The distributed computing environment update scheduling system monitors whether there are any updates to the functions provided to the application. When it detects an update to a function, it determines whether jobs running and jobs waiting on multiple computer nodes are compatible enough to run successfully in the updated computer node environment in order to apply the update to the function. Based on the determination result of whether the running and waiting jobs are compatible, the distributed computing environment update scheduling system generates an execution schedule for the update tasks on the multiple computer nodes.

Description

Technical Field

[0004] , , , , , , , , , , , ,

[0005] ,

[0003]

[0001] The present invention relates to a distributed computing environment update system and a distributed computing environment update method.

Background Art

[0002] In recent years, since machine learning processing (ML processing, particularly model learning) deals with a large amount of data, it is generally executed using a distributed computing environment composed of multiple nodes. Technologies related to machine learning processing are evolving rapidly day by day, and the update frequency of the environment for executing applications, such as applications, libraries, and hardware, has become extremely high. These updates are essential for improving the accuracy of the generated learning model and realizing new functions.

[0003] However, when updating an application or the execution environment, there is a problem of compatibility between the application and the execution environment, and there is a risk that the operation of the entire system becomes unstable. Particularly in a distributed computing environment, since multiple nodes and components are interdependent, it is likely to be difficult to find the cause when a failure occurs. Therefore, there is a need for a mechanism to update the execution environment in accordance with the evolution of ML technology while appropriately ensuring the compatibility between the application and the execution environment so that such failures do not occur.

[0004] Therefore, for example, Patent Document 1 discloses a method for ensuring the compatibility of applications in a distributed computing environment by dynamically checking the compatibility of libraries during software version upgrades or cluster configurations and updating the execution environment while maintaining stability.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

[0006] However, with the technology described in Patent Document 1 mentioned above, if the execution environment is updated frequently, machine learning processing that takes a long time (e.g., more than 3 days) will be frequently stopped or interrupted by the updates, significantly extending the time until the entire process is completed. Thus, it has been difficult to update the environment appropriately and efficiently during the operation of a distributed computing environment.

[0007] This invention has been made in consideration of the above points, and aims to appropriately and efficiently update the environment during the operation of a distributed computing environment. [Means for solving the problem]

[0008] To achieve the above objective, the present invention, in one aspect, is a distributed computing environment update scheduling system that generates a schedule for executing an update task on a plurality of computer nodes to apply an update to the functionality provided to the application, the update task being executed on the plurality of computer nodes to update the environment of the plurality of computer nodes to which a job consisting of tasks related to an application executed by a computer is assigned and executed, wherein the processor of the distributed computing environment update scheduling system monitors whether or not the functionality has been updated, and when it detects the update of the functionality, it determines whether the jobs currently running and the jobs waiting on the plurality of computer nodes are compatible so that they can be executed normally in the updated environment, and generates an execution schedule for the update task on the plurality of computer nodes based on the result of the determination of whether the jobs currently running and the jobs waiting on the computer nodes are compatible. [Effects of the Invention]

[0009] According to the present invention, for example, the environment can be updated appropriately and efficiently during the operation of a distributed computing environment. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing an example of the configuration of a distributed computing system according to the embodiment. [Figure 2] A diagram illustrating the overview of the distributed computing environment update scheduling process in a distributed computing system according to the embodiment. [Figure 3] A diagram showing an example of the hardware configuration of a computer according to this embodiment. [Figure 4] A flowchart illustrating an example of the distributed computing environment update process according to the embodiment. [Figure 5] A flowchart illustrating an example of the update task priority execution node selection process according to the embodiment. [Figure 6] A flowchart showing an example of the update task execution process according to the embodiment. [Figure 7] A diagram illustrating the scheduling of update tasks according to the embodiment. [Figure 8] A flowchart showing an example of the new job scheduling process according to the embodiment. [Figure 9] A diagram illustrating the scheduling of a new task according to the embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings.

[0012] (Configuration of the distributed computing system S according to the embodiment) Figure 1 shows an example of the configuration of a distributed computing system S according to the embodiment. The distributed computing system S generates a schedule for executing update tasks to update the environments of nodes 63 and 64, to which jobs consisting of application-related tasks are assigned and executed, in order to apply updates to the functions provided to the application.

[0013] The distributed computing system S includes a distributed computing environment update scheduling system 1, computer systems 2 and 3, ML (machine learning processing) application external function repositories 4 and 5, a distributed computing environment 6, and an environment information management unit 7. In the distributed computing system S, the distributed computing environment update scheduling system 1, computer systems 2 and 3, ML application external function repositories 4 and 5, distributed computing environment 6, and environment information management unit 7 are connected to the network N.

[0014] The distributed computing environment update scheduling system 1 includes an update monitoring unit 11, a compatibility determination unit 12, an update schedule generation unit 13, and an environment update completion notification unit 14.

[0015] The computer systems 2 and 3 are computers each having a processor such as a CPU that executes an ML application 21 or 31. The ML applications 21 and 31 cause ML tasks 63tk and 64tk that constitute an ML job related to the ML processing instructed by the user to be executed on nodes 63 and 64 of the distributed computing environment 6 to perform machine learning and generate a learning model. The ML applications 21 and 31 are an example of an application that causes the distributed computing environment 6 to execute a task.

[0016] The ML application external function repositories 4 and 5 store modules of external functions provided to the ML applications executed in the distributed computing environment 6.

[0017] The distributed computing environment 6 includes a queuing system 61, a load balancer 62, and nodes 63, 64. The queuing system 61 is a queuing system capable of managing the scheduling of tasks such as Kueue. The queuing system 61 controls the execution schedule of tasks by registering tasks in the queue in the order of execution. The queuing system 61 can change the execution order by assigning priorities to jobs. That is, after the queuing system 61 schedules tasks by default, it can assign priorities to the running tasks and the waiting tasks and review the schedule.

[0018] In addition to the ML tasks 63tk, 64tk, the queuing system 61 also registers update tasks that update the external functions provided from the ML application external function repositories 4, 5 to the ML applications 21, 31 when applied to the nodes 63, 64.

[0019] The nodes 63, 64 are computer nodes having processors such as CPUs that execute one or more ML tasks 63tk, 64tk and update tasks according to the execution schedule controlled by the queuing system 61.

[0020] The environment information management unit 7 acquires and manages environment information including the configuration of the distributed computing environment 6, the execution status of the ML tasks 63tk, 64tk and the update tasks in each of the nodes 63, 64, and the like.

[0021] In this embodiment, two computer systems 2, 3 and two ML application external function repositories 4, 5 are shown, but the number of computer systems and ML application external function repositories is not limited to two. Also, in this embodiment, two nodes 63, 64 are shown, but the number of nodes is not limited to two.

[0022] (Summary of the processing of the embodiment) Figure 2 shows an overview of the distributed computing environment update scheduling process in the distributed computing system S according to the embodiment.

[0023] The update monitoring unit 11 periodically accesses the ML application external function repositories 4 and 5 to check whether there are any updates to the external functions in the ML application external function repositories 4 and 5.

[0024] The compatibility determination unit 12 refers to the queuing system 61 of the distributed computing environment 6 and determines the compatibility of the running and waiting ML tasks scheduled by the queuing system 61 on a job-by-job basis. "Compatible with the execution environment before the execution of the update task" indicates that the nodes 63 and 64, before the execution of the update task that updates the external functions of the ML application, have an execution environment that can successfully execute ML tasks 63tk and 64tk. "Compatible with the execution environment after the execution of the update task" indicates that the nodes 63 and 64, after the execution of the update task that updates the external functions of the ML application, have an execution environment that can successfully execute ML tasks 63tk and 64tk.

[0025] The update schedule generation unit 13 generates an execution schedule for update tasks that update the execution environment at each node 63 and 64, based on the compatibility determination results of the running and waiting ML tasks 63tk and 64tk determined by the compatibility determination unit 12. The update schedule generation unit 13 schedules the update tasks while maintaining a balance of resources, taking into account the status of the running and waiting ML tasks, resource load, and the compatibility of the ML tasks with the updated environment.

[0026] The update schedule generation unit 13 then notifies the queuing system 61 of the generated execution schedule. The queuing system 61 stores the update tasks in a queue according to the execution schedule of the update tasks generated by the update schedule generation unit 13.

[0027] The environment update completion notification unit 14 instructs the environment information management unit 7 to notify computer systems 2 and 3 of the environment update completion notification when the environment update of the distributed computing environment 6 is successfully completed by the execution of update tasks at each node 63 and 64.

[0028] ML applications 21 and 31 instruct the queuing system 61 to execute tasks (ML tasks) of the ML applications. The queuing system 61 assigns ML tasks 63tk, 64tk and update tasks by storing them in queues for each node 63 and 64 in the execution order according to the task execution instructions from ML applications 21 and 31.

[0029] Load balancer 62 monitors the progress of ML tasks 63tk, 64tk and update tasks, and equalizes resource load as needed. If the resource utilization balance deteriorates, the load balancer changes the distribution of ML tasks 63tk and 64tk to other nodes while monitoring the completion status of update tasks on each node 63 and 64 and the resource load balance. When changing the distribution of ML tasks 63tk and 64tk to other nodes, the load balancer may interrupt to ensure that the order does not become slower than the original order.

[0030] Each node 63 and 64 sequentially retrieves and executes the assigned ML tasks 63tk, 64tk, and update tasks from the queue.

[0031] The distributed computing environment 6 manages environmental information, including the configuration of the distributed computing environment 6, and the execution status of ML tasks 63tk, 64tk, and update tasks on each node 63, 64. The distributed computing environment 6 passes environmental information to the environmental information management unit 7 upon request from the environmental information management unit 7. When the computer systems 2, 3 are notified that the environment update of nodes 63, 64 has been completed by the execution of the update task, developers can code ML applications 21, 31 to utilize the updated functions.

[0032] (Computer 300 hardware) Figure 3 shows an example of the hardware configuration of computer 300 according to the embodiment. Computer 300 implements the distributed computing environment update scheduling system 1, computer systems 2 and 3, and nodes 63 and 64 by executing a predetermined program.

[0033] The computer 300 comprises a processor 301 including a CPU, a memory 302, a storage device 303, an input device 304, an output device 305, and a communication IF 306, all interconnected via an internal communication line 307 such as a bus.

[0034] The processor 301 controls the overall operation of the computer 300 and, in cooperation with the memory 302, executes predetermined programs to realize each functional unit. The memory 302 is composed of, for example, volatile semiconductor memory and is used as the work memory of the processor 301. The storage device 303 consists of one or more large-capacity non-volatile storage devices such as hard disk drives, SSDs (Solid State Drives), or flash memory, and is used to retain various programs and data for long periods of time. Executable programs stored in the storage device 303 are loaded into the memory 302 when the computer 300 starts up or when needed, and are executed by the processor 301.

[0035] The executable program may be recorded on a non-temporary recording medium and read from the non-temporary recording medium by a media reader and loaded into memory 302. Alternatively, the executable program may be obtained from an external computer via a network and loaded into memory 302.

[0036] The input device 304 consists of a keyboard, a pointing device such as a mouse, and is used by the user to input various instructions and information into the computer 300. The output device 305 consists of a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker, and is used to present necessary information to the user when needed.

[0037] Communication I / F306 is an interface device for connecting computer 300 to various networks within the system or for communicating with other computers. Communication I / F306 consists of, for example, a NIC (Network Interface Card) such as a wired LAN (Local Area Network) or a wireless LAN.

[0038] (Distributed computing environment update process according to the embodiment) Figure 4 is a flowchart showing an example of the distributed computing environment update process according to the embodiment. The distributed computing environment update process is executed whenever an update to the external functions of the ML applications used by the ML applications 21 and 31 of the computer systems 2 and 3 is detected.

[0039] First, in step S11, the update monitoring unit 11 detects updates to the external functions of the ML application in the ML application external function repositories 4 and 5. Next, in step S12, the update monitoring unit 11 notifies the user of the update to the external functions (execution of an update task) in the distributed computing environment 6.

[0040] Next, in step S13, the compatibility determination unit 12 checks the compatibility of the running job (ML tasks 63tk, 64tk included in the running job) and the standby job (ML tasks 63tk, 64tk included in the standby job) using the method disclosed in Patent Document 1. For example, the compatibility determination unit 12 assigns a compatible label to the jobs that it determined to be compatible in step S13 on a job-by-job basis.

[0041] Next, in step S14, the update schedule generation unit 13 executes the update task priority execution node selection process. Details of the update task priority execution node selection process will be described later with reference to Figure 5.

[0042] Next, in step S15, the update schedule generation unit 13 executes the update task execution process. Details of the update task execution process will be described later with reference to Figure 6.

[0043] Next, in step S16, the environment update completion notification unit 14 outputs a notification command to the environment information management unit 7 indicating that the environment update is complete. The environment information management unit 7 periodically acquires environment information from the distributed computing environment 6 and obtains the execution status of update tasks at each node 63, 64.

[0044] Next, in step S17, the environmental information management unit 7 determines whether the update task has been completed at all nodes 63 and 64. If the update task has been completed at all nodes 63 and 64 (step S17YES), the environmental information management unit 7 proceeds to step S18. On the other hand, if the update task has not been completed at all nodes 63 and 64 (step S17NO), the environmental information management unit 7 repeats step S17.

[0045] In step S18, the environmental information management unit 7 notifies computer systems 2 and 3 that the update tasks on all nodes 63 and 64 have been completed.

[0046] (Selection process for priority execution node of update task according to the embodiment) Figure 5 is a flowchart showing an example of the update task priority execution node selection process according to the embodiment. The update task priority execution node selection process is called in step S14 of the distributed computing environment update process (Figure 4). The update task priority execution nodes are nodes 63 and 64 on which the update task is executed without waiting for the completion of other running and waiting jobs.

[0047] First, in step S141, the update schedule generation unit 13 estimates the number of nodes required for each job based on the model size (number of parameters) of the learning model to be generated and information on the resource status of nodes 63 and 64. For example, regarding the resource status, it is assumed that each node 63 or 64 has 2 CPUs, 2 GPUs, and 40GB of memory. For example, if the model size of the learning model to be generated is 80B, the number of required nodes = number of model parameters / amount of memory per GPU = 80 / 20 = 4 nodes. Also, for example, if the model size of the learning model to be generated is 60B, the number of required nodes = number of model parameters / amount of memory per GPU = 60 / 20 = 3 nodes.

[0048] Next, in step S142, the update schedule generation unit 13 determines the proportion of nodes that will be given priority in executing update tasks. That is, it determines the proportion of nodes that will be given priority in executing update tasks based on the number of nodes required by compatible jobs that are running and waiting, and the number of nodes required by incompatible jobs that are running and waiting. For example, if the number of nodes required by compatible jobs is 2 and the number of nodes required by incompatible jobs is 3, the proportion of nodes that will be given priority in executing update tasks will be 2 / (2+3)=2 / 5.

[0049] Next, in step S143, the update schedule generation unit 13 selects nodes from among those currently executing compatible jobs that have compatible jobs scheduled for them as update task priority execution nodes, up to a number determined from the ratio of nodes determined in step S142. The update schedule generation unit 13 then tags the information related to the update task priority execution nodes. For example, if the number of update task priority execution nodes does not reach the number determined from the ratio of nodes determined in step S142, it randomly selects nodes with incompatible jobs scheduled to reach this number.

[0050] (Update task execution process according to the embodiment) Figure 6 is a flowchart showing an example of the update task execution process according to the embodiment. The update task execution process is executed in parallel on all nodes of the distributed computing environment 6.

[0051] First, in step S151, the node (either node 63 or 64) determines whether the corresponding node in the distributed computing environment 6 has been tagged with the tag that was determined to be a “priority execution node for update tasks” in step S143 (Figure 5). If the tag is found to be attached (step S151 YES), the update schedule generation unit 13 moves the process to step S157 for the corresponding node. On the other hand, if the tag is not found to be attached (step S151 NO), the update schedule generation unit 13 moves the process to step S152 for the corresponding node.

[0052] In step S152, the node continues executing the jobs that are currently running on that node. Then, in step S153, since the execution of the jobs in step S152 has finished, the node accepts and executes any pending incompatible jobs.

[0053] Next, in step S154, the node determines whether there are any pending incompatible jobs to accept after the execution of the job in step S153 has finished. If there are no pending incompatible jobs to accept next (step S154 YES), the node moves processing to step S155. On the other hand, if there are pending incompatible jobs to accept next (step S154 NO), the node returns processing to step S153.

[0054] In step S155, the node executes the update task. Then, in step S156, the node accepts and executes a waiting compatible job. Meanwhile, in step S157, the node prioritizes executing the update task.

[0055] (Overview of scheduling update tasks according to the embodiment) Figure 7 shows an overview of the scheduling of update tasks according to the embodiment. In Figure 7, "(a) Before scheduling update tasks" shows that the ML task for job J1 of application 1, which is compatible with the updated environment in which the update task was executed, is running on nodes 1 to 3. Also in Figure 7, "(a) Before scheduling update tasks" shows that the ML task for job J3 of application 3, which is compatible with the updated environment, is waiting on nodes 1 to 4. Also in Figure 7, "(b) After scheduling update tasks" shows that the ML task for job J2 of application 2, which is not compatible with the updated environment, is running on nodes 4 to 7.

[0056] ML tasks compatible with the updated environment should ideally be run in the updated environment, considering the improved accuracy of the models produced.

[0057] Therefore, as shown in "(a) Before scheduling the update task" in Figure 7, the execution of compatible ML tasks included in job J2 of application 2, which is running on nodes 4 to 7, is interrupted. Then, as shown in "(b) After scheduling the update task" in Figure 7, job J2 is queued and waiting on nodes 4 to 7 so that it will be re-executed after the update task is completed.

[0058] Furthermore, ML tasks that are incompatible with the updated environment should ideally be run in the pre-update environment, considering the improvement in the accuracy of the generated models.

[0059] Therefore, as shown in "(a) Before scheduling the update task" in Figure 7, the scheduling of ML task J3tk, which is waiting on node 4 and is an incompatible ML task included in job J3 of application 3, is changed. That is, as shown in "(b) After scheduling the update task" in Figure 7, all ML tasks of job J3 are queued and waiting on nodes 1 to 3 so that they will be executed after the update task is completed. Then, all waiting ML tasks of job J3 are queued and waiting so that the update task will be executed after they have finished executing.

[0060] The distributed computing environment update scheduling system 1 optimizes and generates execution schedules so that running jobs (tasks), waiting jobs (tasks), and update tasks are executed at appropriate times, and the load on nodes 1 to 7 is leveled.

[0061] Furthermore, the GUI (Graphical User Interface) displaying Figure 7 can also be output from a display device connected to the distributed computing environment update scheduling system 1. This allows the user to understand the status of compatible and incompatible tasks before and after scheduling the update task, as well as the scheduling status of the update task and any changes thereto.

[0062] (New job scheduling process according to the embodiment) Figure 8 is a flowchart showing an example of the new job scheduling process according to the embodiment. The new job scheduling process is executed when a new job arrives before the update task is completed at all nodes 63 and 64. Based on the execution status of existing ML tasks 63tk and 64tk at nodes 63 and 64, including whether they are compatible or not, the new job scheduling process ensures that compatible or incompatible ML tasks 63tk and 64tk are executed consecutively. In principle, jobs containing ML tasks 63tk and 64tk executed at each node 63 and 64 will follow the execution order of "incompatible → update task → compatible".

[0063] First, in step S21, the update schedule generation unit 13 determines the compatibility of the new job. Next, in step S22, the update schedule generation unit 13 determines whether the compatibility determination result of step S21 for the new job is correct. If the new job is correct (step S22 YES), the update schedule generation unit 13 moves the process to step S23. On the other hand, if the new job is incorrect (step S22 NO), the update schedule generation unit 13 moves the process to step S24.

[0064] In step S23, the update schedule generation unit 13 schedules a new task to either node 63 or 64 where the update task has been completed. Meanwhile, in step S24, the update schedule generation unit 13 asks the user via the GUI whether to modify the new job's tasks to ensure compatibility or to execute them on the computer nodes 63 or 64 in the environment before the update, and determines whether the user has been instructed to modify the tasks. If the user has been instructed to modify the new job's tasks to ensure compatibility (step S24 YES), the update schedule generation unit 13 moves to step S25. On the other hand, if the user has been instructed to execute the new job's tasks on the computer nodes 63 or 64 in the environment before the update (step S24 NO), the update schedule generation unit 13 moves to step S26.

[0065] In step S25, the update schedule generation unit 13 modifies the relevant task to ensure compatibility. Once step S25 is completed, the update schedule generation unit 13 moves the process to step S23.

[0066] In step S26, the update schedule generation unit 13 schedules a new task to either node 63 or 64, where no update task has been executed. The new job assigned to either node 63 or 64 in steps S23 and S26 is executed on each node 63 or 64 according to the schedule.

[0067] (Overview of scheduling new tasks according to the embodiment) Figure 9 shows an overview of the scheduling of a new task according to this embodiment.

[0068] As illustrated in Figure 9, the new ML tasks tk1, tk2, and tk3 are checked for compatibility with the updated environment. The compatible ML task tk2 is queued and waiting to run on one of the nodes in the updated environment.

[0069] Furthermore, if a node's environment is being updated, the user will be notified that the environment of that node is being updated. In addition, if new ML tasks tk1 and tk3 are incompatible with the updated environment, the user will be asked whether or not to modify the incompatible new ML tasks to make them compatible. Depending on the user's selection, ML task tk3 will be modified into the compatible ML task tk3-1, and ML task tk3-1 will be queued and waiting to be executed on the updated node, for example, node 4. On the other hand, ML task tk1, which was not modified to be compatible, will be queued and waiting to be executed as an interrupt before the update task, for example, node 1.

[0070] Furthermore, the GUI displaying Figure 9 can also be output from a display device connected to the distributed computing environment update scheduling system 1. This allows users to understand the scheduling status of new tasks.

[0071] (Effects of the embodiment) In the above embodiment, an execution schedule for update tasks is generated across multiple computing nodes based on the determination results of whether running and pending jobs are compatible. Therefore, environment updates can be efficiently performed without terminating tasks while the distributed computing environment is in operation. Furthermore, the inefficiency of running tasks compatible with the updated environment in the pre-update environment before running them in the updated environment can be avoided.

[0072] Furthermore, in the above-described embodiment, a preferred execution computer node is selected to interrupt the running job and execute the update task based on the determination result of whether the running job and the waiting job are compatible. Therefore, by performing the environment update on the preferred execution computer node where the environment update is possible, the execution of compatible jobs in the appropriate environment can be started quickly.

[0073] Furthermore, in the above embodiment, a priority execution computer node is selected to interrupt running jobs and execute update tasks based on the proportion of the required number of nodes. Therefore, a priority execution computer node can be appropriately selected depending on the compatibility of running and waiting jobs and the resource status.

[0074] In the above embodiment, on computer nodes other than the priority execution computer node, the execution of jobs currently running continues, and once the execution of the currently running jobs is completed, other incompatible jobs are executed. Then, once all incompatible jobs have finished executing, the update task is executed, and once the execution of the update task is completed, other compatible jobs are executed. Thus, each task can be executed sequentially in the order of incompatible tasks → update task → compatible tasks.

[0075] Furthermore, in the above-described embodiment, tasks included in incompatible jobs that were assigned to the preferred execution computer node are reassigned to computer nodes other than the preferred execution computer node. Therefore, each task can be executed sequentially in the order of incompatible task → update task → compatible task.

[0076] Furthermore, in the above embodiment, it is determined whether a new task that arises while an update task is being executed on multiple computer nodes is compatible, and an execution schedule for the new task on the multiple computer nodes is generated based on the result of the determination of whether the new task is compatible. Therefore, even while an update task is being executed, the new task can be appropriately assigned to any of the multiple computer nodes.

[0077] In the above embodiment, if a new task is incompatible, the user is asked whether to modify the new task to make it compatible. Upon receiving the user's instruction to modify the new task to make it compatible, the new task is modified to make it compatible, and an execution schedule is generated so that the modified new task runs on the computer node after the update task has finished running. On the other hand, upon receiving the user's instruction not to modify the new task to make it compatible, an execution schedule is generated so that the update task runs on the computer node where the update task has not yet been executed. Therefore, new tasks that arise during the execution of an update task can be executed in either the updated environment after the update task has finished running or the pre-update environment where the update task has not yet been executed, according to the user's instruction, and the progress of the job including the task can be completed quickly.

[0078] Furthermore, in the above-described embodiment, the computer is notified that the update of the computing node's environment has been completed by the execution of the update task. Therefore, developers can quickly utilize the latest updated features when coding applications.

[0079] Furthermore, in the above-described embodiment, execution schedules for running jobs, waiting jobs, and update tasks are generated to equalize the load across multiple computing nodes. This improves the utilization efficiency of the distributed computing environment and, consequently, the development efficiency of applications such as ML applications.

[0080] Furthermore, in the above-described embodiment, the application is an application that requires a long processing time, such as an ML application that performs machine learning processing. Therefore, it is possible to avoid the inconvenience of the application's processing being frequently stopped or interrupted due to updates to the execution environment, and to prevent a significant delay in the time it takes to complete the entire process.

[0081] The present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail for the purpose of clearly illustrating the present invention, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations. In addition, some or all of the above-described configurations, functions, processing units, processing means, etc., may be realized in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc., may be realized in software by a processor interpreting and executing a program that realizes each function. Such a program may be acquired via a non-temporary medium or acquired from an external device via a communication line and installed in a computer equipped with a processor. [Explanation of symbols]

[0082] S: Distributed computing system, 1: Distributed computing environment update scheduling system, 2,3: Computer system, 6: Distributed computing environment, 11: Update monitoring unit, 12: Compatibility determination unit, 13: Update schedule generation unit, 14: Environment update completion notification unit, 21,31: ML application, 61: Queuing system, 62: Load balancer, 63,64: Node, 300: Computer, 301: Processor, 302: Memory

Claims

1. A distributed computing environment update scheduling system that generates a schedule for executing update tasks on multiple computer nodes to apply updates to the functionality provided to an application, where jobs consisting of tasks related to an application run by a computer are assigned and executed on the environments of multiple computer nodes. The processor of the aforementioned distributed computing environment update scheduling system is The system monitors whether the aforementioned function has been updated, When an update to the aforementioned function is detected, it is determined whether the jobs currently running and jobs waiting on the multiple computer nodes are compatible and can be executed successfully in the updated environment. Based on the determination result of whether the jobs currently running and the jobs waiting are compatible, the execution schedule for the update task is generated on the multiple computer nodes. A distributed computing environment update scheduling system characterized by the following:

2. A distributed computing environment update scheduling system according to claim 1, The aforementioned processor, Based on the determination result of whether the jobs currently running and the jobs waiting are compatible, a preferred execution computer node is selected from among the plurality of computer nodes to interrupt the running jobs and execute the update task. A distributed computing environment update scheduling system characterized by the following:

3. A distributed computing environment update scheduling system according to claim 2, The aforementioned processor, Based on the resource status of each of the aforementioned computer nodes, the number of computer nodes required to execute the jobs currently running and the jobs waiting is calculated. The ratio of the required number of nodes for each of the jobs that are compatible and those that are not, among the jobs currently running and the jobs that are on standby, which are assigned to the plurality of computer nodes, is calculated. The preferred execution computer node is selected based on the proportion of the required number of nodes. A distributed computing environment update scheduling system characterized by the following:

4. A distributed computing environment update scheduling system according to claim 2, The aforementioned processor, On the computer nodes other than the designated priority execution computer node, the execution of the currently running job will continue, and once the execution of the currently running job is complete, the other incompatible job will be executed. Once all the incompatible jobs have finished executing, the update task will be executed. Once the aforementioned update task is completed, the other compatible jobs will be executed. A distributed computing environment update scheduling system characterized by the following:

5. A distributed computing environment update scheduling system according to claim 2, The aforementioned processor, Tasks included in the incompatible jobs that were assigned to the preferred execution computer node will be reassigned to computer nodes other than the preferred execution computer node. A distributed computing environment update scheduling system characterized by the following:

6. A distributed computing environment update scheduling system according to claim 1, The aforementioned processor, It is determined whether any new tasks that arise during the execution of the update tasks on the aforementioned plurality of computer nodes have the aforementioned compatibility. Based on the determination result of whether the new task is compatible, the execution schedule for the new task is generated on the multiple computer nodes. A distributed computing environment update scheduling system characterized by the following:

7. A distributed computing environment update scheduling system according to claim 6, The aforementioned processor, If the new task does not have the aforementioned compatibility, the user will be asked whether to modify the new task to make it compatible. Upon receiving instructions from the user to modify the new task to ensure compatibility, the system modifies the new task to ensure compatibility and generates an execution schedule so that the modified new task runs on the computer node after the update task has finished running. Upon receiving instructions from the user that they will not modify the new task to maintain the aforementioned compatibility, the system generates an execution schedule so that the update task is executed on the computer node where it has not yet been executed. A distributed computing environment update scheduling system characterized by the following:

8. A distributed computing environment update scheduling system according to claim 1, The computer is notified that the update of the environment of the computer node has been completed by the execution of the update task. A distributed computing environment update scheduling system characterized by the following:

9. A distributed computing environment update scheduling system according to claim 1, The aforementioned processor, To equalize the load across the multiple computer nodes, the execution schedules for the running jobs, the waiting jobs, and the update tasks are generated. A distributed computing environment update scheduling system characterized by the following:

10. A distributed computing environment update scheduling system according to claim 1, The aforementioned application is an application that performs machine learning processing. A distributed computing environment update scheduling system characterized by the following:

11. A distributed computing environment update scheduling method executed by a distributed computing environment update scheduling system that generates a schedule for executing update tasks on multiple computer nodes to apply updates to the functionality provided for an application, the update tasks being executed on multiple computer nodes to update the environment of the multiple computer nodes to which jobs consisting of tasks related to an application run by a computer are assigned and executed, The processor of the aforementioned distributed computing environment update scheduling system The system monitors whether the aforementioned function has been updated, When an update to the aforementioned function is detected, it is determined whether the jobs currently running and jobs waiting on the multiple computer nodes are compatible and can be executed successfully in the updated environment. Based on the determination result of whether the jobs currently running and the jobs waiting are compatible, the execution schedule for the update task is generated on the multiple computer nodes. A method for scheduling updates to a distributed computing environment, characterized by having each of the following processes.

Images