Information processing program, information processing device, and information processing method

Abstract

To enable efficient use of computing resources. [Solution] If the sum of the amount of context information stored in the memory of the first computing resource 10b-1 to 10b-3, which is one of the multiple computing resources 10b-1 to 10b-3 and has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource 10b-1 to 10b-3 is suppressed.

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing apparatus, and an information processing method. [Background Art]

[0002] It is known that processing performance can be improved by using a GPU (Graphics Processing Unit) instead of a CPU (Central Processing Unit) to execute a deep learning application (hereinafter referred to as a deep learning app).

[0003] Since the unit price of a GPU is higher than that of a CPU, it is important to properly share and use a small number of GPUs among a plurality of programs (processes).

[0004] Furthermore, GPU preemption is known as an example of a technique for efficiently using GPUs. In GPU preemption, it is possible to externally stop a task that is using a GPU and transfer the right to use the GPU to another task. By periodically performing such GPU preemption, GPU usage tasks can be switched in units of time, and subsequent tasks can use the GPU without waiting until the preceding task is completely stopped. [Prior Art Literature] [Patent Literature]

[0005] [Patent Literature 1] Japanese Unexamined Patent Application Publication No. 2010-287254 [Patent Literature 2] U.S. Patent No. 8253749 Specification [Patent Literature 3] Japanese Unexamined Patent Application Publication No. 2013-25823 [Summary of the Invention] [Problem to be Solved by the Invention]

[0006] Many AI (Artificial Intelligence) programs reserve context information for using the GPU in the GPU's memory (GPU memory) until the program terminates. Context information includes, for example, information about the program and the GPU. In GPU memory, the size of this context information can reach around 1GB. The memory area where this context information is stored can be called the context memory area.

[0007] Context information is removed from GPU memory when the program terminates, so it remains in GPU memory until the program is terminated. Furthermore, running multiple programs in a multi-GPU environment, and repeatedly moving programs between GPUs, can increase the context memory area on at least some GPUs, potentially straining GPU memory. If there is insufficient free GPU memory, program execution may be hindered.

[0008] In one aspect, the present invention aims to enable the efficient use of computing resources. [Means for solving the problem]

[0009] This information processing program, when assigning a target program to one of the multiple computing resources capable of executing the target program in a computer having multiple computing resources, causes the computer to execute a process to suppress the assignment of the target program to the first computing resource if the sum of the amount of context information stored in the memory of the first computing resource, which has not yet executed the target program, and the amount of context information for the target program is greater than or equal to a first threshold. [Effects of the Invention]

[0010] According to one embodiment, computing resources can be used efficiently. [Brief explanation of the drawing]

[0011] [Figure 1] This diagram schematically shows the configuration of an information processing system according to one embodiment. [Figure 2] This is a block diagram showing an example of a computer hardware (HW) configuration that realizes the functions of an information processing system according to one embodiment. [Figure 3] This figure illustrates a method for acquiring information about management parameters in an information processing system according to one embodiment. [Figure 4] This figure illustrates a method for determining whether a GPU can be moved by the scheduler unit of an information processing system according to one embodiment. [Figure 5] This figure illustrates the relaxed conditions for determining whether a GPU is available or unavailable by the scheduler unit of an information processing system according to one embodiment. [Figure 6] This is a flowchart illustrating the overview of the GPU allocation process in an information processing system according to one embodiment. [Figure 7] This is a flowchart illustrating an example of the GPU movement feasibility determination process performed by the scheduler unit of an information processing system according to one embodiment. [Figure 8] This is a flowchart illustrating an example of backfill processing by the scheduler unit of an information processing system according to one embodiment. [Modes for carrying out the invention]

[0012] Embodiments relating to this information processing program, information processing apparatus, and information processing method will be described below with reference to the drawings. However, the embodiments shown below are merely illustrative, and there is no intention to exclude various modifications or applications of technologies not explicitly shown in the embodiments. In other words, these embodiments can be implemented with various modifications without departing from their spirit. Furthermore, each figure is not intended to represent only the components shown in the figure, but may include other functions, etc.

[0013] (A) Configuration FIG. 1 is a diagram schematically illustrating the configuration of an information processing system 1 according to an embodiment, and FIG. 2 is a block diagram illustrating a hardware (HW) configuration example of a computer 10 that implements functions of the information processing system 1 according to the embodiment.

[0014] (A-1) Hardware Configuration Example When a plurality of computers are used as HW resources that implement the functions of the information processing system 1, each computer may have the HW configuration exemplified in FIG. 2.

[0015] As illustrated in FIG. 2, the computer 10 is an information processing apparatus, and may illustratively include, as a HW configuration, one or more (one in the example illustrated in FIG. 2) CPUs 10a, two or more (three in the example illustrated in FIG. 2) GPUs 10b-1 to 10b-3, a memory 10c, a storage unit 10d, an IF (Interface) unit 10e, an IO (Input / Output) unit 10f, and a reading unit 10g. Hereinafter, when the GPUs 10b-1 to 10b-3 are not particularly distinguished, they are referred to as a GPU 10b.

[0016] The CPU 10a is an example of an arithmetic processing device that performs various controls and calculations, and is a control unit that executes various processes. The CPU 10a may be communicatively connected to each block in the computer 10 via a bus 10j. The bus 10j may include a PCIe (Peripheral Component Interconnect-Express) bus. Note that the CPU 10a may be a multiprocessor including a plurality of processors, may be a multi-core processor having a plurality of processor cores, or may have a configuration including a plurality of multi-core processors.

[0017] The GPU 10b may be, for example, an accelerator such as a GPGPU (General Purpose computing on Graphics Processing Unit). Furthermore, the GPU 10b may be used to perform screen display control for an output device such as a monitor in the IO unit 10f. The GPU 10b may be configured as an accelerator that executes machine learning processing and inference processing using a machine learning model. It can be said that the GPU 10b has higher processing performance than the CPU 10a with respect to machine learning processing and inference processing. The GPU 10b is an example of a plurality of computing resources capable of executing an assigned processing target program. The computer 10 is an example of a computer having a plurality of computing resources (CPU 10a, GPU 10b). The CPU 10a and the plurality of GPUs 10b are an example of a plurality of computing resources.

[0018] Furthermore, a VRAM (Video Random Access Memory), which is not shown in the figure, is mounted on the GPU 10b. The VRAM of the GPU 10b may be referred to as a GPU memory. The GPU memory is an example of a second memory used by the GPU 10b (the second computing resource). The GPUs 10b-1 to 10b-3 may have the same performance, or at least some of them may have different performances. In the present embodiment, an example in which the GPUs 10b-1 to 10b-3 have the same performance will be described.

[0019] These CPU 10a and GPUs 10b-1 to 10b-3 are computing resources allocated to a user program 105 which will be described later. Furthermore, GPU 10b-1 may be referred to as GPU#1, GPU 10b-2 as GPU#2, and GPU 10b-3 as GPU#3 in some cases.

[0020] Memory 10c is an example of hardware that stores various data and program information. Examples of memory 10c include volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory), or both. Memory 10c is an example of first memory used by the CPU 10a. Memory 10c can be referred to as CPU memory.

[0021] The storage unit 10d is an example of hardware that stores various data and program information. Examples of storage units 10d include magnetic disk devices such as HDDs (Hard Disk Drives), semiconductor drive devices such as SSDs (Solid State Drives), and various storage devices such as non-volatile memory. Examples of non-volatile memory include flash memory, SCM (Storage Class Memory), and ROM (Read Only Memory).

[0022] The memory unit 10d may store a program 10h (information processing program) that implements all or part of the various functions of the computer 10.

[0023] For example, the CPU 10a of the information processing system 1 realizes the functions of a GPU allocation unit 101 and a deep learning framework 106 by loading the program 10h stored in the memory unit 10d into memory 10c and executing it. The functions of the GPU allocation unit 101 and the deep learning framework 106 can be called scheduling functions.

[0024] The IF unit 10e is an example of a communication interface that controls the connection and communication between this computer 10 and other computers. For example, the IF unit 10e may include an adapter compliant with LAN (Local Area Network) such as Ethernet®, or optical communication such as FC (Fibre Channel). The adapter may support wireless, wired, or both communication methods. The program 10h may be downloaded from the network to the computer 10 via the communication interface and stored in the storage unit 10d.

[0025] The I / O unit 10f may include either an input device or an output device, or both. Examples of input devices include keyboards, mice, and touch panels. Examples of output devices include monitors, projectors, and printers. The I / O unit 10f may also include a touch panel or the like that integrates the input and output devices. The output device may be connected to the GPU 10b. The I / O unit 10f may also be an input or output device of another information processing device remotely connected to this computer 10 via SSH (Secure Shell) or the like.

[0026] The reading unit 10g is an example of a reader that reads data and program information recorded on the recording medium 10i. The reading unit 10g may include a connection terminal or device to which the recording medium 10i can be connected or inserted. Examples of the reading unit 10g include an adapter compliant with USB (Universal Serial Bus), a drive device for accessing a recording disk, and a card reader for accessing flash memory such as an SD card. The recording medium 10i may store a program 10h, and the reading unit 10g may read the program 10h from the recording medium 10i and store it in the storage unit 10d.

[0027] Examples of recording media 10i include non-temporary computer-readable recording media such as magnetic / optical discs and flash memory. Examples of magnetic / optical discs include flexible discs, CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray discs, and HVDs (Holographic Versatile Discs). Examples of flash memory include semiconductor memory such as USB memory and SD cards.

[0028] The hardware configuration of computer 10 described above is illustrative. Therefore, the addition or deletion of hardware within computer 10 (for example, adding or deleting arbitrary blocks), division, integration in any combination, or addition or deletion of buses may be performed as appropriate.

[0029] (A-2) Example of Functional Configuration As shown in Figure 1, the information processing system 1 may, for example, include functions as a GPU assigner 101, a user program 105, and a deep learning framework 106. These functions may be implemented by the hardware of the computer 10 (see Figure 2).

[0030] User program 105 may be a program that implements processes such as training a deep learning model (machine learning model) or performing inference using a deep learning model, and may execute jobs related to deep learning. User program 105 is, for example, a deep learning program.

[0031] In AI processing, similar processes are repeatedly executed, as seen in batch processing during the learning process and inference processing on inference servers. For example, in the deep learning (training phase) process, preprocessing, machine learning (main process), and postprocessing are repeatedly executed. Similarly, in the inference phase process, preprocessing such as receiving requests, inference (main process), and postprocessing are repeatedly executed.

[0032] CPU 10a executes the processing related to the allocation of user program 105. In this process, CPU 10a assigns the pre-processing and post-processing of user program 105 to itself (CPU 10a), and assigns the main processing of user program 105 to GPU 10b. Hereafter, for convenience, the processing performed by CPU 10a executing user program 105 will be expressed as if user program 105 were the main entity, for example, "user program 105 performs the processing".

[0033] The user program 105 may, for example, output an allocation request to the scheduler unit 102 of the GPU allocation unit 101 to switch computing resources from CPU 10a to GPU 10b when transitioning from preprocessing to main processing. This allocation request to switch computing resources from CPU 10a to GPU 10b may be called a GPU request.

[0034] Furthermore, the user program 105 may output an allocation request to the scheduler unit 102 to switch computing resources from the GPU 10b to the CPU 10a when transitioning from the end of this process to post-processing.

[0035] A series of processes including pre-processing, main processing, and post-processing may be referred to as a job. In this information processing system 1, one user program 105 may execute one process, and multiple processes may be executed by executing multiple user programs 105. Alternatively, one user program 105 may execute multiple processes. In this embodiment, for simplicity, an example in which one user program 105 executes one process is shown. Also, below, the user program 105 may simply be referred to as a program.

[0036] In this information processing system 1, the GPU allocation unit 101, described later, performs scheduling to allocate computing resources (CPU 10a, GPU 10b) to each of multiple programs and execute them.

[0037] Furthermore, when the user program 105 has finished using the computing resources, it may output a request to release the computing resources to the scheduler unit 102.

[0038] For example, when user program 105 completes processing using GPU 10b, it sends a GPU release notification to scheduler unit 102. Also, when user program 105 completes processing using CPU 10a, it sends a CPU release notification to scheduler unit 102.

[0039] Furthermore, the user program 105 receives the request results that the scheduler unit 102 responds to (returns) in response to various requests such as GPU allocation requests, CPU allocation requests, GPU release requests, and other GPU release requests that the user program 105 has sent to the scheduler unit 102.

[0040] Deep Learning Framework 106 is a framework that performs processing related to deep learning.

[0041] The deep learning framework 106 is software that functions as the base for the user program 105 and is provided in conjunction with the user program 105. The user program 105 is executed on top of this deep learning framework 106. Therefore, a deep learning framework 106 may be provided for each user program 105.

[0042] The deep learning framework 106 is foundational software for efficiently advancing machine learning using the user program 105, and may include, for example, commonly used processing patterns in the user program 105 as a library. The deep learning framework 106 causes the user program 105 to run on the GPU 10b. In other words, the deep learning framework 106 utilizes the GPU 10b.

[0043] Furthermore, when the deep learning framework 106 executes the user program 105 on the GPU 10b, it measures the amount of GPU memory used for each GPU 10b and sends the measurement results to the program management unit 103 of the GPU allocation unit 101.

[0044] The deep learning framework 106 may measure the amount of GPU memory used in response to a request from the GPU allocation unit 101 and transmit the measurement results to the program management unit 103. Alternatively, the deep learning framework 106 may autonomously measure the amount of GPU memory used at various predetermined timings and transmit the measurement results to the program management unit 103.

[0045] The GPU allocation unit 101 assigns the program to the GPU 10b. As shown in Figure 1, the GPU allocation unit 101 also functions as a scheduler unit 102, a program management unit 103, and a GPU management unit 104.

[0046] The GPU management unit 104 manages the state of the GPU memory of the GPUs 10b. In the example shown in Figure 1, the GPU management unit 104 manages the free memory status of each GPU 10b using the free memory management information 25. In the free memory management information 25 exemplified in Figure 1, the size of the free memory area of ​​each GPU is associated with each of the GPUs #1 to #3.

[0047] Furthermore, the GPU management unit 104 manages the data size (size of the context memory area) of the context information in the GPU memory of each GPU 10b using context memory management information 26. In the context memory management information 26 exemplified in Figure 1, the size of the context information stored in the respective GPU memory is associated with each of the GPUs #1 to #3. It can be said that the GPU management unit 104 manages the context memory area of ​​the GPU memory of each GPU 10b using context memory management information 26.

[0048] The GPU management unit 104 may, for example, query the scheduler unit 102 (described later) periodically or irregularly at predetermined intervals to obtain the size of the free GPU memory area and the size of the context information for each GPU 10b. The GPU management unit 104 registers the size of the free GPU memory area and the size of the context information, as responded by the scheduler unit 102, into the free memory management information 25 and the context memory management information 26.

[0049] The program management unit 103 manages information about programs executed by the GPU 10b. In the example shown in Figure 1, the program management unit 103 uses memory usage management information 20 to manage the size of GPU memory (memory usage) used by each program when it is executed by the GPU 10b. This memory usage includes the size of the context information. In the memory usage management information 20 exemplified in Figure 1, the respective memory usage is associated with each of the programs #0 to #2.

[0050] Furthermore, the program management unit 103 manages the data size (size of the context memory area) of the context information for each program using the context memory size management information 21. The data size of the context information for each program is managed for each GPU 10b. In the context memory size management information 21 exemplified in Figure 1, the size of the context information is associated for each GPU 10b for each of programs #0 to #2.

[0051] Furthermore, the program management unit 103 manages the execution time per iteration of each program using the execution time management information 22. The execution time per iteration represents the time taken to execute each iteration (GPU execution time) when the GPU 10b is made to execute the program. In the execution time management information 22 illustrated in Figure 1, the time taken by the GPU 10b to process one iteration is associated with each of the programs #0 to #2. In this embodiment, an example is shown in which GPUs #1 to #3 have the same performance, but if the performance of GPUs #1 to #3 differs, the execution time management information 22 may manage the GPU execution time for each GPU 10b.

[0052] Furthermore, the program management unit 103 uses the remaining iteration count management information 23 to manage the number of iterations that have not yet been executed in each program (remaining iteration count). In the remaining iteration count management information 23 illustrated in Figure 1, the remaining iteration count is associated with each of the programs #0 to #2.

[0053] Furthermore, the program management unit 103 uses the moveability management information 24 to manage whether each program can be moved between GPUs. Some programs cannot be moved between GPUs due to constraints. For example, programs that handle multiple GPUs may not be able to move between GPUs due to synchronization and parameter sharing relationships. In the moveability management information 24 illustrated in Figure 1, information indicating whether or not a program can be moved between GPUs (allowed / not allowed) is associated with each of programs #0 to #2.

[0054] The program management unit 103 may, for example, query the deep learning framework 106 at predetermined intervals, either periodically or irregularly, for information such as the memory usage, context information size, GPU execution time per iteration, and remaining iterations for the user program 105. The program management unit 103 registers the information received from the deep learning framework 106 into the memory usage management information 20, context memory size management information 21, execution time management information 22, and remaining iteration count management information 23.

[0055] The program management unit 103 and the GPU management unit 104 acquire values ​​for multiple types of management parameters based on various measurements taken by the deep learning framework 106.

[0056] Figure 3 is a diagram illustrating a method for acquiring information about management parameters in an information processing system 1 according to one embodiment.

[0057] Figure 3 shows an example of executing the main process of program #0 (see symbol P1) and the main process of program #1 (see symbol P2) on GPU #1. In Figure 3, each main process indicated by symbols P1 and P2 represents one iteration.

[0058] The deep learning framework 106 measures the time taken to execute each iteration (GPU execution time: execution time per iteration) when running the program on the GPU 10b (see symbol A1 in Figure 3). The measured execution time per iteration is stored in the execution time management information 22.

[0059] Furthermore, the deep learning framework 106 measures the amount of context memory used by the program (see symbol A2 in Figure 3). The deep learning framework 106 measures the amount of GPU memory used after the GPU 10b has executed one iteration of the program for the first time and has finished execution (the amount of memory used by the program after the GPU 10b has finished using the program), and uses this measurement result as the amount of context memory used by the program. The deep learning framework 106 measures the amount of context memory for each program. The measured amount of context memory used by the program is stored in the context memory management information 21.

[0060] Furthermore, the deep learning framework 106 measures the amount of context memory on the GPU 10b (see symbol A3 in Figure 3). The deep learning framework 106 uses the sum of the context memory amounts of one or more programs that have run on the GPU 10b (provided they have not terminated) as the amount of context memory on the GPU 10b. The measured (calculated) amount of context memory on the GPU 10b is stored in the context memory management information 26.

[0061] Furthermore, the deep learning framework 106 measures the amount of GPU memory used for each program (see symbol A4 in Figure 3). In addition, the user may input information regarding the maximum number of iterations for a program and whether or not the program can be moved, and the scheduler unit 102, etc., may acquire the input values ​​(see symbol A5 in Figure 3).

[0062] The scheduler unit 102 schedules the execution of the user program 105 on each GPU 10b.

[0063] The scheduler unit 102, for example, uses a waiting queue (not shown) to allocate programs stored in the waiting queue to the GPU 10b using a FIFO (First In First Out) method.

[0064] For example, when the scheduler unit 102 receives a GPU request from a user program 105, it registers the program in the waiting queue.

[0065] The scheduler unit 102 may refer to the free memory management information 25 managed by the GPU management unit 104 and select a GPU 10b with available free memory for program allocation as the allocation destination for the program. Note that when all GPUs 10b are unused, such as during startup of this information processing system 1, all GPUs 10b are available for use as program allocation destinations.

[0066] When a GPU 10b becomes available, the scheduler unit 102 assigns the program at the top of the waiting queue to the available GPU 10b.

[0067] Furthermore, the scheduler unit 102 controls the program being executed on one of the multiple GPUs 10b (second computing resource: for example, GPU#1) to be moved to another available GPU 10b (first computing resource: for example, GPU#2), and to execute it on the destination GPU 10b. This improves the utilization efficiency of the GPUs 10b.

[0068] In this manner, when a program (the program to be processed) is moved between GPUs 10b and executed on the destination GPU 10b, the source GPU 10b is an example of a second computing resource, and the destination GPU 10b is an example of a first computing resource. Note that program movement between GPUs 10b can be achieved using known methods, and their explanation is omitted here.

[0069] Furthermore, the scheduler unit 102 controls the movement of programs between GPUs 10b by restricting the movement of programs between GPUs 10b before the amount of context memory on a GPU 10b affects the execution of the program on that GPU 10b, thereby suppressing an increase in the amount of context memory.

[0070] The scheduler unit 102 determines whether the program can be moved if it does not use a GPU 10b that has been used in the past (used GPU 10b) as the destination for the program to be moved, that is, if the candidate GPU 10b for the destination of the program to be moved is a GPU 10b that has not been used to execute the program in the past. In other words, the scheduler unit 102 determines whether the program to be moved can be moved if the context information of the program to be moved is not stored in the candidate GPU 10b for the destination. The candidate GPU 10b for the destination is an example of the first computing resource.

[0071] The following process of determining whether a program running on another GPU10b (the program to be moved) can be moved to a candidate GPU10b can be referred to as GPU move feasibility determination.

[0072] In determining whether a GPU can be moved, the scheduler unit 102 determines that the program to be moved can be moved to the candidate GPU 10b if the candidate GPU 10b satisfies the following condition (1).

[0073] (GPU context memory amount + program context memory amount) <Threshold for movable context memory amount...Condition (1) In the above condition (1), "GPU context memory amount" is the context memory amount of the candidate destination GPU10b, and is an example of context information stored in the memory (GPU memory) of the first computing resource (candidate destination GPU10b) among multiple computing resources (GPU10b) that has not yet executed the program to be processed.

[0074] "Program context memory amount" refers to the context memory amount of the program being moved, and is an example of the amount of context information of the program being processed. "Threshold for movable context memory amount" is an example of the first threshold.

[0075] If the above condition (1) is not met, the scheduler unit 102 determines that the program to be moved cannot be moved to the candidate GPU 10b and suppresses the allocation of the program to be processed to the candidate GPU 10b (first computing resource). In other words, the scheduler unit 102 suppresses the allocation of the program to be processed to the candidate GPU 10b (first computing resource) if the sum of the "GPU context memory amount" and the "program context memory amount" is greater than or equal to the "threshold for movable context memory amount (first threshold)".

[0076] The threshold for the amount of movable context memory may be set appropriately depending on the system's usage and the characteristics of the program.

[0077] The scheduler unit 102 obtains the amount of context memory of the candidate GPU 10b to move to from the context memory management information 26, and obtains the amount of context memory of the program to be moved from the context memory amount management information 21. Based on these values, the scheduler unit 102 determines whether condition (1) is met.

[0078] Furthermore, if the moveability management information 24 indicates that the program to be moved cannot be moved between GPUs 10b, that is, if the program to be moved cannot move GPUs due to program constraints or the like, the scheduler unit 102 also determines that the program to be moved cannot be moved to the candidate destination GPU 10b, and suppresses the allocation of the program to be processed to the candidate destination GPU 10b (first computing resource).

[0079] Figure 4 is a diagram illustrating the method used by the scheduler unit 102 of the information processing system 1 according to one embodiment to determine whether or not a GPU can be moved.

[0080] In Figure 4, the context memory amount for GPU#1 and GPU#2, calculated as "GPU context memory amount + program context memory amount," is shown in shaded areas, and the threshold for movable context memory is shown with a dashed line.

[0081] In the example shown in Figure 4, the value of "GPU context memory amount + program context memory amount" for GPU#1 is greater than or equal to the threshold for movable context memory amount. Therefore, since GPU#1 does not satisfy the above condition (1), the scheduler unit 102 determines that it is not possible to move the program to GPU#1.

[0082] On the other hand, the value of "GPU context memory amount + program context memory amount" for GPU#2 is smaller than the threshold for the amount of movable context memory. Therefore, since GPU#2 satisfies the above condition (1), the scheduler unit 102 determines that it is possible to move the program to GPU#2.

[0083] Furthermore, when the scheduler unit 102 determines whether a GPU can be moved, even if the above-mentioned condition (1) is not met, it determines that the program to be moved can be moved to the candidate GPU 10b if the following condition (2) is met, and at least one of the following conditions (3) and (4) is also met. In other words, the scheduler unit 102 relaxes the GPU move feasibility determination if condition (2) is met, and at least one of the following conditions (3) and (4) is also met.

[0084] (GPU context memory amount + program context memory amount) <(GPU memory amount - maximum memory usage by programs on the GPU) ... Condition (2) In condition (2) above, "GPU context memory amount" refers to the context memory amount of the candidate destination GPU 10b, and "program context memory amount" refers to the context memory amount of the program to be moved. Also, "GPU memory amount" is the size of the GPU memory storage area. "Maximum memory usage by the program on the GPU" is the maximum amount of memory used by the program running on GPU 10b, which can be obtained, for example, from the deep learning framework 106.

[0085] This condition (2) indicates that the total amount of context memory on the GPU when the target program is moved to the candidate destination GPU10b is less than the amount of remaining memory after subtracting the area used by the program from the total GPU memory.

[0086] In other words, the scheduler unit 102 relaxes the GPU migration feasibility determination and suppresses the execution of suppression of the allocation of the program to be processed to the candidate GPU 10b (first computing resource) if there is free space in the memory (GPU memory) of the candidate GPU 10b (first computing resource) to store the context information of the program to be processed.

[0087] (Execution time per program iteration × Number of remaining iterations) <(Threshold for program execution time) ···Condition (3) Condition (3) indicates that the time required for the program to be moved to complete the remaining iterations is less than or equal to the threshold. The threshold for the time required for moveable execution may be set as appropriate depending on the system's usage and the program's characteristics.

[0088] The scheduler unit 102 relaxes the GPU transfer feasibility determination and suppresses the execution of the suppression of assigning the program to the candidate destination GPU 10b (first computing resource) when the remaining execution time of the program to be processed is less than the threshold for program transferable execution time (second threshold).

[0089] (Number of remaining program iterations) < threshold ... Condition (4) This condition (4) indicates that the remaining iterations of the program are less than a threshold (third threshold). The scheduler unit 102 relaxes the GPU migration feasibility determination and suppresses the execution of suppressing the allocation of the program to the candidate destination GPU 10b (first computing resource) when the number of remaining iterations of the program to be processed is less than the third threshold (third threshold = 2 in this embodiment), that is, when the number of remaining iterations of the program = 1.

[0090] Conditions (3) and (4) both indicate that the remaining execution time of the program to be moved is short and that it will terminate soon. When a program terminates, its context information is removed from GPU memory. Therefore, even if condition (1) is not met, if either condition (3) or (4) is met, the remaining execution time of the program to be moved is short, and it is expected that condition (1) will be met soon. In other words, even if a program is moved between GPU memories, it is expected that the execution of the program on the destination GPU 10b will not be interrupted.

[0091] Figure 5 is a diagram illustrating the relaxed conditions for determining whether a GPU is usable by the scheduler unit 102 of the information processing system 1 according to one embodiment.

[0092] In Figure 5, the "GPU context memory amount + program context memory amount" for GPU#1 and GPU#2 are shaded. The "maximum memory usage by the program on the GPU" is indicated by diagonal lines. Furthermore, the threshold for movable context memory is indicated by a dashed line.

[0093] In the example shown in Figure 5, the value of "GPU context memory amount + program context memory amount" is greater than the "threshold for movable context memory amount" for both GPU #1 and #2. Therefore, neither GPU #1 nor #2 satisfies the above condition (1) (see shaded area).

[0094] However, in the case of GPU#1, even after subtracting "GPU context memory amount + program context memory amount" and "maximum memory usage by the program on the GPU" from "GPU memory amount," there is still free GPU memory (see the white-filled area). In other words, the above condition (2) is satisfied. It is considered that programs that terminate quickly can be processed in this free GPU memory area shown as the white-filled area.

[0095] In this information processing system 1, even if condition (1) is not met, if condition (2) is met, it is permitted to move and execute a program that can be completed in a short time from another GPU 10b. In other words, the determination of whether or not a GPU can be moved based on condition (1) is relaxed.

[0096] (B) Operation The outline of the GPU 10b allocation process in the information processing system 1 according to one embodiment configured as described above will be explained according to the flowchart (steps A1 to A10) shown in Figure 6.

[0097] In step A1, the scheduler unit 102 obtains the program at the head of the waiting queue (the target program).

[0098] In step A2, the scheduler unit 102 checks whether the number of available GPUs 10b (number of available GPUs) is greater than or equal to the number of GPUs 10b required to run the target program (number of required GPUs ≤ number of available GPUs).

[0099] If the number of available GPUs is greater than or equal to the required number of GPUs (see the Yes route in step A2), proceed to step A3. In step A3, the scheduler unit 102 checks whether the available GPU 10b confirmed in step A2 is a used GPU 10b that has been used to run the target program (used GPU ⊂ available GPU).

[0100] If the available GPU10b is the same as the used GPU10b (see the Yes route in step A3), proceed to step A10.

[0101] In step A10, the scheduler unit 102 responds to the user program 105 by assigning the used GPU 10b to the target program. Then, the process returns to step A1.

[0102] If the results of the check in step A3 show that the available GPU10b is not the used GPU10b (see No. root in step A3), proceed to step A4.

[0103] In step A4, the scheduler unit 102 obtains information about available GPUs 10b from the GPU management unit 104, and in step A5, it obtains information about the target program from the program management unit 103. Then, in step A6, the scheduler unit 102 determines whether a GPU move is possible. That is, the scheduler unit 102 determines whether an available GPU 10b can be used as a destination for a program currently running on another GPU 10b. Details of the process in step A6 will be described later using the flowchart shown in Figure 7.

[0104] Subsequently, in step A7, the scheduler unit 102 checks whether the number of movable GPUs 10b (number of movable GPUs) is greater than or equal to the number of required GPUs 10b (number of required GPUs ≤ number of movable GPUs). If the number of movable GPUs is greater than or equal to the number of required GPUs (see the Yes route in step A7), the process proceeds to step A10. In step A10, the scheduler unit 102 responds to the user program 105 by assigning the movable GPUs 10b to the target program. After that, the process returns to step A1.

[0105] If the results of the check in step A7 indicate that the number of movable GPUs is less than the number of required GPUs (see No. route in step A7), proceed to step A8. Also, if the results of the check in step A2 indicate that the number of available GPUs is less than the number of required GPUs (see No. route in step A2), proceed to step A8.

[0106] In step A8, the scheduler unit 102 performs a backfill process. In this backfill process, while the program at the front of the waiting queue waits for the required number of GPUs 10b to be secured, GPUs 10b are allocated to other programs to fill the available GPUs 10b. Details of the process in step A8 will be described later using the flowchart shown in Figure 8.

[0107] Subsequently, in step A9, the scheduler unit 102 waits until enough free GPUs 10b become available to meet the number required for the execution of the target program. If more than the number of free GPUs 10b required for the execution of the target program become available, the process proceeds to step A10.

[0108] In step A10, the scheduler unit 102 responds to the user program 105 by assigning an available GPU 10b to the target program as GPU 10b. Then, the process returns to step A1.

[0109] Next, an example of the GPU movement feasibility determination process by the scheduler unit 102 of the information processing system 1 according to one embodiment will be explained according to the flowchart (steps B1 to B9) shown in Figure 7. This process shows the details of step A6 of the flowchart shown in Figure 6.

[0110] In step B1, the scheduler unit 102 checks whether there is a GPU 10b that has been previously used to execute the program to be moved (a used GPU 10b) among the available GPUs 10b.

[0111] If a used GPU 10b exists among the available GPUs 10b (see the Yes route in step B1), in step B2, the scheduler unit 102 starts determining whether or not to move the GPU to the available GPU 10b (the candidate destination GPU 10b).

[0112] In step B3, the scheduler unit 102 checks whether the above condition (1) is met for the target GPU 10b (destination candidate GPU 10b) extracted from the available GPUs 10b. That is, the scheduler unit 102 checks whether the "context memory amount of the target GPU 10b + context memory amount of the program to be moved" is less than the threshold for the amount of movable context memory.

[0113] If condition (1) is met (see the Yes route in step B3), in step B4, the scheduler unit 102 determines that the program to be moved can be moved to the target GPU 10b.

[0114] Subsequently, in step B9, the scheduler unit 102 checks if there is an available GPU 10b for which a GPU move eligibility determination has not yet been made. If there is an available GPU 10b for which a GPU move eligibility determination has not yet been made, the process returns to step B3. If there is no available GPU 10b for which a GPU move eligibility determination has not yet been made, the process ends. After that, the process proceeds to step A7 of the flowchart in Figure 6.

[0115] Furthermore, if, as a result of the check in step B3, condition (1) is not met for the target GPU 10b (target candidate GPU 10b) (see No. route in step B3), the process proceeds to step B6. In step B6, the scheduler unit 102 checks whether the above condition (3) is met for the target GPU 10b (target candidate GPU 10b). That is, the scheduler unit 102 checks whether "execution time per iteration of the program to be moved × number of remaining iterations" is less than the threshold for the program's movable execution time.

[0116] If condition (3) is met (see the Yes route in step B6), the process proceeds to step B7. In step B7, the scheduler unit 102 checks whether the above condition (2) is met for the target GPU 10b (destination candidate GPU 10b). That is, the scheduler unit 102 checks whether "the amount of context memory of the destination candidate GPU 10b + the amount of context memory of the program to be moved" is less than "the amount of GPU memory of the destination candidate GPU 10b - the maximum amount of memory used by the program executed on the destination candidate GPU 10b".

[0117] If condition (2) is met (see the Yes route in step B7), the process proceeds to step B4. That is, the scheduler unit 102 determines that the program to be moved can be moved to the target GPU 10b.

[0118] If condition (3) is not met in step B6 (see No. route in step B6), proceed to step B8. Also, if condition (2) is not met in step B7 (see No. route in step B7), proceed to step B8.

[0119] In step B8, the scheduler unit 102 determines that the program to be moved cannot be moved to the target GPU 10b. The process then proceeds to step B9.

[0120] Furthermore, if the verification in step B1 indicates that there are no used GPU10b among the available GPU10b (see No. root in step B1), proceed to step B5.

[0121] In step B5, the scheduler unit 102 determines that the program to be moved can be moved to all GPUs 10b. After that, the process proceeds to step B9.

[0122] In the example shown in Figure 7, the scheduler unit 102 does not check whether condition (4) is met, but it is not limited to this. The scheduler unit 102 may, for example, determine whether condition (4) is met instead of processing in step B6, or it may determine whether condition (4) is also met in addition to processing in step B6, and so on. It can be implemented in various modified ways.

[0123] Next, an example of backfill processing by the scheduler unit 102 of the information processing system 1 according to one embodiment will be explained according to the flowchart (steps C1 to C12) shown in Figure 8. This process shows the details of step A8 of the flowchart shown in Figure 6.

[0124] In step C1, the scheduler unit 102 copies the programs stored in the waiting queue to the backfill queue (not shown) in ascending order of the value "execution time per iteration × number of GPUs". As a result, the scheduler unit 102 performs the processing in steps C2 to C8, described later, on the programs stored in the waiting queue, starting with the programs that can be processed the fastest, thereby reducing the amount of context memory used.

[0125] In step C2, the scheduler unit 102 checks the information of the program at the head of the backfill queue. For example, the scheduler unit 102 checks the number of GPUs required by the program.

[0126] In step C3, the scheduler unit 102 checks if there is an available GPU 10b. If there is no available GPU 10b (see No route in step C3), this process is terminated. After that, the process proceeds to step A9 in the flowchart of Figure 6.

[0127] On the other hand, if there are available GPUs 10b (see the Yes route in step C3), the process proceeds to step C4. In step C4, the scheduler unit 102 checks whether the number of available GPUs is greater than or equal to the number of GPUs 10b required to run the target program (required GPUs ≤ number of available GPUs). If the number of available GPUs is less than the required GPUs (see the No route in step C4), the process returns to step C2.

[0128] On the other hand, if the number of available GPUs is greater than or equal to the required number of GPUs (see the Yes route in step C4), the process proceeds to step C5. In step C5, the scheduler unit 102 checks whether the available GPU 10b confirmed in step C3 is a used GPU 10b that has been used to run the target program (used GPU ⊂ available GPU). If there are multiple available GPUs 10b, it is checked whether all of these available GPUs 10b are used GPUs 10b. If the available GPU10b is not the used GPU10b (see No root in step C5), proceed to step C6.

[0129] In step C6, the scheduler unit 102 obtains information about available GPUs 10b from the GPU management unit 104, and in step C7, it obtains information about the target program from the program management unit 103. Then, in step C8, the scheduler unit 102 determines whether a GPU move is possible. That is, the scheduler unit 102 determines whether an available GPU 10b can be used as a destination for a program currently running on another GPU 10b. The details of the process in step C8 are the same as those shown in the flowchart in Figure 7.

[0130] Subsequently, in step C9, the scheduler unit 102 checks whether the number of movable GPUs is greater than or equal to the number of required GPUs (required GPUs ≤ number of movable GPUs). If the number of movable GPUs is less than the number of required GPUs (see No. route in step C9), the process returns to step C2.

[0131] On the other hand, if the number of movable GPUs is greater than or equal to the number of required GPUs (see the Yes route in step C9), proceed to step C10. Also, if the result of the check in step C5 is that the available GPU 10b is already in use (see the Yes route in step C5), proceed to step C10.

[0132] In step C10, the scheduler unit 102 checks whether the scheduled start time of the first program in the waiting queue will be delayed. This is to prevent the start of processing of the first program in the waiting queue from being delayed by the backfill process. The scheduled start time of the first program in the waiting queue can be determined using known scheduling methods, so the explanation is omitted.

[0133] If the scheduled start time for the first program in the waiting queue is delayed (see No route in step C10), return to step C2.

[0134] On the other hand, if the scheduled start time of the first program in the waiting queue is not delayed (see the Yes route in step C10), in step C11, the scheduler unit 102 responds to the user program 105 by assigning the used GPU 10b as the GPU 10b to the target program. Then, in step C12, the scheduler unit 102 checks whether it has performed a backfill feasibility check (processing in steps C2 to C11) for all programs in the backfill queue. If there are programs for which a backfill feasibility check has not been performed (see the No route in step C12), it returns to step C2. Also, if it has performed a backfill feasibility check for all programs in the backfill queue (the Yes route in step C12), it terminates the process. Then, it proceeds to step A9 in the flowchart of Figure 6.

[0135] (C) Effects Thus, according to the information processing system 1 as an example of an embodiment, the scheduler unit 102 can efficiently execute the program by moving the program to be moved between GPUs 10b and executing it on the destination GPU 10b (high scheduling efficiency).

[0136] Furthermore, by restricting the movement of programs between GPUs 10b, it is possible to suppress the increase in the amount of context memory in GPU memory, thereby reducing the occurrence of program execution failures and scheduling failures.

[0137] Furthermore, if the above condition (1) is not met, the scheduler unit 102 determines that the program to be moved cannot be moved to the candidate destination GPU 10b, and suppresses the allocation of the program to be processed to the candidate destination GPU 10b (first computing resource). In this way, by restricting program movement between GPUs 10b when condition (1) is not met, program execution failures can be suppressed.

[0138] Furthermore, the scheduler unit 102 restricts program movement between GPUs 10b, thereby suppressing an increase in the amount of context memory on the GPUs 10b. This allows for a larger amount of free memory in the GPU memory, thereby suppressing program execution failures caused by insufficient free memory.

[0139] Furthermore, by having the scheduler unit 102 restrict the movement of programs between GPUs 10b, the occurrence of idle GPUs 10b can be suppressed, thereby improving the utilization efficiency of the GPUs 10b.

[0140] Furthermore, by performing backfill processing, the utilization efficiency of GPU10b can be improved without increasing the amount of context memory. In addition, by prioritizing the allocation of programs that terminate quickly during this backfill processing, the increase in the amount of context memory in GPU memory can be suppressed.

[0141] Furthermore, the scheduler unit 102 relaxes the GPU transfer eligibility determination if condition (2) above is met, and at least one of conditions (3) and (4) is also met. That is, for programs that are to be moved and have a short remaining execution time and are expected to finish soon, the restrictions on movement between GPUs 10b are relaxed. Since context information is deleted when the program finishes, this makes it possible to make effective use of the available GPUs 10b and improve the utilization efficiency of the GPUs 10b.

[0142] The program cannot be moved to GPU10b, which has a large amount of context memory, and a backfill process is performed to run the program that has been using GPU10b previously. This suppresses the increase in the amount of context memory in GPU10b's GPU memory, and allows the program to run stably on GPU10b.

[0143] By combining the relaxation of program movement restrictions between GPU10b and backfill processing, it is possible to suppress the increase in the amount of context memory in GPU memory, thereby mitigating its impact and improving the utilization efficiency of GPU10b.

[0144] (D) Other The disclosed technology is not limited to the embodiments described above and can be implemented in various modified forms without departing from the spirit of this embodiment.

[0145] For example, the embodiments described above show an example where the computing resource is a GPU, but it is not limited to this. The computing resource may be a processor other than a GPU, etc.

[0146] Furthermore, the above disclosure makes it possible for those skilled in the art to implement and manufacture this embodiment.

[0147] (E) Note The following additional information is disclosed regarding the embodiments described above.

[0148] (Note 1) In a computer with multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the multiple computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing program characterized by causing the computer to perform the processing.

[0149] (Note 2) If the remaining execution time of the program to be processed is less than the second threshold, the execution of the process to be suppressed will be suppressed. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0150] (Note 3) If the remaining number of iterations of the program to be processed is less than the third threshold, the execution of the process to be suppressed will be suppressed. The information processing program described in Appendix 1, characterized in that it causes the computer to perform the processing.

[0151] (Note 4) If there is free space in the memory of the first computing resource to store the context information of the program to be processed, Suppression of the execution of the aforementioned suppression process An information processing program according to Appendix 2 or 3, characterized in that it causes the computer to perform the processing.

[0152] (Note 5) When moving the program to be processed, which is currently running on the second computing resource among the multiple computing resources, to the first computing resource, the allocation of the program to be processed is performed. An information processing program according to any one of the appendices 1 to 3, characterized in that it causes the computer to perform the processing.

[0153] (Note 6) An information processing device having multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the multiple computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing apparatus characterized by having a control unit that performs processing.

[0154] (Note 7) The control unit, If the remaining execution time of the program to be processed is less than the second threshold, the execution of the process to be suppressed will be suppressed. The information processing apparatus according to Appendix 6, characterized by performing processing.

[0155] (Note 8) The control unit, If the remaining number of iterations of the program to be processed is less than the third threshold, the execution of the process to be suppressed will be suppressed. The information processing apparatus according to Appendix 6, characterized by performing processing.

[0156] (Note 9) The control unit, If there is free space in the memory of the first computing resource to store the context information of the program to be processed, Suppression of the execution of the aforementioned suppression process An information processing apparatus according to appendix 7 or 8, characterized by performing processing.

[0157] (Note 10) The control unit, When moving the program to be processed, which is currently running on the second computing resource among the multiple computing resources, to the first computing resource, the allocation of the program to be processed is performed. An information processing device according to any one of the appendices 6 to 8, characterized by performing processing.

[0158] (Note 11) In a computer with multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the multiple computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing method characterized in that the processing is performed by the computer.

[0159] (Note 12) If the remaining execution time of the program to be processed is less than the second threshold, the execution of the process to be suppressed will be suppressed. The information processing method according to Appendix 11, characterized in that the processing is performed by the computer.

[0160] (Note 13) If the remaining number of iterations of the program to be processed is less than the third threshold, the execution of the process to be suppressed will be suppressed. The information processing method according to Appendix 11, characterized in that the processing is performed by the computer.

[0161] (Note 14) If there is free space in the memory of the first computing resource to store the context information of the program to be processed, the execution of the process to be suppressed is suppressed. The information processing method according to Appendix 12 or 13, characterized in that the processing is performed by the computer.

[0162] (Note 15) When moving the program to be processed, which is currently running on the second computing resource among the multiple computing resources, to the first computing resource, the allocation of the program to be processed is performed. The information processing method according to any one of the appendices 11 to 13, characterized in that the processing is performed by the computer. [Explanation of Symbols]

[0163] 1. Information Processing System 10 Computers 10a CPU 10b-1~10b-3, 10b GPU 10c memory 10d storage section 10e IF section 10f IO section 10g reading unit 10-hour program 10i recording media 10j bus 20. Memory Usage Management Information 21. Context Memory Management Information 22. Execution Time Management Information 23. Remaining Iteration Count Management Information 24 Movement permission management information 25. Free Memory Management Information 26 Context Memory Management Information 101 GPU allocation section 102 Scheduler Section 103 Program Management Department 104 GPU management department 105 User Programs 106 Deep Learning Frameworks

Claims

1. In a computer with multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the plurality of computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing program characterized by causing the computer to perform the processing.

2. If the remaining execution time of the program to be processed is less than the second threshold, the execution of the process to be suppressed will be suppressed. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

3. If the remaining number of iterations of the program to be processed is less than the third threshold, the execution of the process to be suppressed will be suppressed. The information processing program according to claim 1, characterized in that it causes the computer to perform the processing.

4. If there is free space in the memory of the first computing resource to store the context information of the program to be processed, Suppression of the execution of the aforementioned suppression process The information processing program according to claim 2 or 3, characterized in that it causes the computer to perform the processing.

5. When moving the program to be processed, which is currently running on the second computing resource among the plurality of computing resources, to the first computing resource, the allocation of the program to be processed is performed. An information processing program according to any one of claims 1 to 3, characterized in that it causes the computer to perform the processing.

6. An information processing device having multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the plurality of computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing apparatus characterized by having a control unit that performs processing.

7. In a computer with multiple computing resources, When assigning the program to be processed to one of the multiple computing resources capable of executing the program to be processed, If the sum of the amount of context information stored in the memory of a first computing resource among the plurality of computing resources that has not yet executed the program to be processed, and the amount of context information of the program to be processed, is greater than or equal to a first threshold, the allocation of the program to be processed to the first computing resource is suppressed. An information processing method characterized in that the processing is performed by the computer.

Images