Memory deallocation program, memory deallocation method, and information processing apparatus.

Abstract

To improve GPU utilization. [Solution] When the GPU 103 is assigned to process the target program 110, the information processing device 101 identifies the available memory on the GPU 103. Based on a comparison of the identified available memory 104 with the memory usage measured during a previous execution of the target program 110, the information processing device 101 determines whether there is insufficient free memory in the memory 104. If there is sufficient free memory in the memory 104, the information processing device 101 executes process 1 of the target program 110 using the GPU 103b. If there is insufficient free memory in the memory 104, the information processing device 101 waits for the GPU 103b to execute process 1 of the target program 110 until a storage area within the memory 104 allocated to another program is released.

Description

Technical Field

[0004]

[0001] The present invention relates to a memory release processing program, a memory release processing method, and an information processing apparatus.

Background Art

[0002] Conventionally, there is a technique for dynamically switching resources for executing an application to improve the resource utilization rate of the entire system. For example, when executing a plurality of programs, programs that are better processed by a GPU (Graphics Processing Unit) and programs that may be processed by a CPU (Central Processing Unit) are distinguished by predicting the acceleration rate or the like, and a GPU is allocated to the processing of a program with a higher priority.

[0003] As a prior art, for example, a host system adds data to an update log on the host memory, reflects the data added to the host-side update log to an update log on the device memory via an I / O bus, and a GPU reflects the data reflected on the update log to a data structure for parallel processing. Also, there is a technique including a VRAM cache module for a graphics processing unit to provide and manage additional cache resources for a central processing unit. Also, there is a technique for transferring the calculation result of a first calculation unit to a second calculation unit using a determined optimal path by determining an optimal path for transferring the calculation result. Also, there is a technique related to memory management for heterogeneous systems such as a system including a CPU and a GPU.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

[0005] However, conventional technologies have a problem in that GPU utilization can decrease when running programs that repeatedly perform processes using machine learning models.

[0006] In one aspect, the present invention aims to improve GPU utilization. [Means for solving the problem]

[0007] In one embodiment, when a GPU shared by two or more programs is assigned to process a target program that is repeatedly executed using a machine learning model, a memory release processing program is provided that identifies the available memory on one of the GPUs, compares the identified available memory with the memory usage measured during a previous execution of the target program's processing, and determines whether there is insufficient available memory. If there is sufficient available memory, the program executes the target program's processing on one of the GPUs; if there is insufficient available memory, the program waits for the GPU to release memory allocated to another program before executing the target program's processing. [Effects of the Invention]

[0008] According to one aspect of the present invention, it has the effect of improving GPU utilization. [Brief explanation of the drawing]

[0009] [Figure 1]Figure 1 is an explanatory diagram showing one embodiment of the memory release processing method according to the embodiment. [Figure 2] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. [Figure 3] Figure 3 is a block diagram showing an example of the hardware configuration of the execution control device 201. [Figure 4] Figure 4 is an explanatory diagram showing an example of the contents of the memory usage management table 220. [Figure 5] Figure 5 is an explanatory diagram showing an example of the contents of the free memory management table 230. [Figure 6] Figure 6 is a block diagram showing an example of the functional configuration of the execution control device 201. [Figure 7] Figure 7 is an explanatory diagram showing an example of the operation of the execution control device 201. [Figure 8] Figure 8 is an explanatory diagram illustrating an example of how to determine whether or not there is insufficient memory. [Figure 9] Figure 9 is an explanatory diagram (part 1) showing an example of data transfer before and after applying this memory release processing method. [Figure 10] Figure 10 is an explanatory diagram (part 2) showing an example of data transfer before and after applying this memory release processing method. [Figure 11] Figure 11 is a flowchart (part 1) showing an example of the memory deallocation procedure of the execution control unit 201. [Figure 12] Figure 12 is a flowchart (part 2) showing an example of the memory deallocation procedure of the execution control unit 201. [Figure 13] Figure 13 is a flowchart showing an example of the execution control processing procedure of the execution control device 201. [Figure 14] Figure 14 is an explanatory diagram showing an example of changes in GPU usage. [Modes for carrying out the invention]

[0010] Hereinafter, with reference to the drawings, embodiments of a memory release processing program, a memory release processing method, and an information processing apparatus according to the present invention will be described in detail.

[0011] (Embodiment) FIG. 1 is an explanatory diagram showing an example of a memory release processing method according to an embodiment. In FIG. 1, the information processing apparatus 101 is a computer that controls the execution of the target program 110. The target program 110 is a program to be executed, and for example, it is a user program.

[0012] The target program 110 includes processing that is repeatedly executed using a machine learning model. The machine learning model is generated by machine learning such as, for example, deep learning. The machine learning model is specified by, for example, an algorithm and parameters (weight parameters).

[0013] The processing of the target program 110 may be, for example, a learning process of a machine learning model that uses all the learning data in a dataset once. Also, the processing of the target program 110 may be an estimation process using a machine learning model (trained model) for each input data group.

[0014] The CPU 102 and the GPU 103 are arithmetic devices that can be used for the execution of the target program 110 and other programs, and are shareable by two or more programs respectively. The CPU 102 and the GPU 103 may be possessed by the information processing apparatus 101, or may be possessed by a computer different from the information processing apparatus 101. There may be two or more CPU 102 and GPU 103 respectively. In the example of FIG. 1, as the GPU 103, two devices, GPU 103a and 103b, are assumed.

[0015] In this context, processing using machine learning models is often better suited to a GPU (e.g., GPU103) than a CPU (e.g., CPU102), and execution on a GPU tends to be faster than execution on a CPU. On the other hand, the number of available GPUs is limited, and when running multiple programs, it may not be possible to allocate a GPU to the processing of all programs.

[0016] Therefore, when running multiple programs, there are conventional scheduling techniques that distinguish between programs that are better processed by the GPU and programs that can be processed by the CPU by measuring the performance of each when executed on the GPU and CPU, and then allocate the GPU to process the programs with higher priority.

[0017] To utilize a GPU, the data required for execution must be moved to the GPU's memory. For example, in a process using a machine learning model, the model data will be moved. Model data is information that represents the machine learning model, and includes information that identifies the machine learning model's algorithm and parameters (weight parameters).

[0018] In the following explanation, the movement of data required for the execution of a program (for example, program 110) may be referred to as "data transfer." Furthermore, the movement of model data within data transfer may be referred to as "model transfer." Additionally, data other than model data required for the execution of a program's processing may be referred to as "input / output data," and the movement of input / output data may be referred to as "input / output data transfer." Input / output data includes, for example, training data sets, input data for machine learning models (trained models), and output data.

[0019] However, when running programs that repeatedly perform processes using machine learning models, the longer the data transfer time, the lower the GPU utilization becomes. GPU utilization is, for example, equivalent to the total data processing time divided by the total time. Data processing time is the time spent executing each process of the program on the GPU. Total time is the time it takes for the entire program to complete execution.

[0020] For example, in conventional technology, memory is released after each execution of a program in order to free up the GPU. Data stored in the GPU's memory is then saved, for example, to the CPU's memory. Therefore, in conventional technology, data transfer occurs each time a program finishes processing.

[0021] For example, let's assume that the process of target program 110 is used as the training process for a machine learning model, and that it is executed three times, switching between datasets. Here, the process executed the first time in target program 110 is referred to as "process 0", the process executed the second time as "process 1", and the process executed the third time as "process 2".

[0022] In this case, with conventional technology, when a GPU is assigned to process 0, a data transfer is first performed to move the data required for executing process 0 to the GPU, and then process 0 is executed on the GPU. When the execution of process 0 is completed, a data transfer is performed to save the data in the GPU's memory to the CPU's memory, and the GPU's memory is freed.

[0023] The data transfer time is the sum of the model transfer time and the input / output data transfer time. The same process as in process 0 is performed for subsequent processes 1 and 2. In processes using large-scale machine learning models, the amount of model data is large, and model transfer becomes a bottleneck when trying to improve GPU utilization.

[0024] Therefore, to improve GPU utilization, it is desirable to reduce the number of model transfers. The model data remains the same across processes that use the machine learning model. For this reason, it is conceivable to release GPU memory when the program starts executing, rather than when the program finishes executing.

[0025] This means that when a GPU is allocated for program processing, if the same GPU is allocated as in the previous execution, the model data in the GPU's memory can be reused, eliminating the need for model transfer. On the other hand, if a different GPU is allocated than in the previous execution, the GPU can be freed up by releasing the memory allocated to the previously allocated GPU.

[0026] However, timing the release of GPU memory to the beginning of program execution may result in situations where sufficient memory cannot be allocated when the GPU is assigned. If memory cannot be allocated to store the data required for program execution despite the GPU being assigned, errors such as out-of-memory may occur, potentially leading to a decrease in GPU utilization efficiency.

[0027] Therefore, in this embodiment, considering the case in which memory capacity for storing data required for execution cannot be secured when allocating a GPU, a memory release processing method for managing memory on the GPU will be described. Here, an example of processing by the information processing device 101 (corresponding to the processing in (1) to (3) below) will be described.

[0028] (1) When the GPU 103 is assigned to process the target program 110, the information processing device 101 identifies the available memory on the GPU 103. Here, we assume that the process of the target program 110 is "process 1" which is executed for the second time, and that the GPU 103b is assigned to process 1. Note that the first process 0 has already been executed with the GPU 103a assigned to it.

[0029] In this case, when the GPU 103b is allocated to process 1 of the target program 110, the information processing device 101 identifies the available memory 104 on the GPU 103b. Here, it is assumed that the storage area 105 within memory 104 is being used by another program, and the available memory 104 is "X [GB]".

[0030] (2) The information processing device 101 compares the available space in the identified memory 104 with the memory usage measured during the previous execution of the target program 110. Based on the results of this comparison, the information processing device 101 determines whether or not there is insufficient free space in the memory 104.

[0031] Here, the memory usage corresponds to the amount of memory required to execute the processing of the target program 110. If memory 104 cannot be allocated a memory area equivalent to this memory usage, it means that there is insufficient free memory in 104, and an error may occur when processing 1 is executed on the GPU 103b.

[0032] Here, let's assume that the memory usage "A [GB]" measured during the first execution of process 0 was recorded. In this case, the information processing device 101 compares the free capacity "X [GB]" of memory 104 with the memory usage "A [GB]". Based on the comparison result, the information processing device 101 determines whether or not there is insufficient free space in memory 104.

[0033] Specifically, for example, the information processing device 101 determines that there is no shortage of free memory in memory 104 if the free capacity "X [GB]" of memory 104 is equal to or greater than the memory usage "A [GB]". On the other hand, if the free capacity "X [GB]" of memory 104 is less than the memory usage "A [GB]", the information processing device 101 determines that there is a shortage of free memory in memory 104.

[0034] Here, we assume that the free space "X [GB]" in memory 104 is less than the memory usage "A [GB]". In this case, the information processing device 101 determines that there is insufficient free space in memory 104.

[0035] (3) If there is sufficient free space in memory 104, the information processing device 101 will execute process 1 of the target program 110 using the GPU 103b. If there is insufficient free space in memory 104, the information processing device 101 will wait for the GPU 103b to execute process 1 of the target program 110 until the memory area in memory 104 allocated to another program is released.

[0036] In this case, there is insufficient free memory 104. In this situation, the information processing device 101 waits for the GPU 103b to execute process 1 of the target program 110 until a storage area within memory 104 (for example, storage area 105) that has been allocated to another program is released.

[0037] Specifically, for example, the information processing device 101 waits to execute process 1 until the memory area in memory 104 is freed and the free space in memory 104 is at least equal to the memory usage amount "A [GB]". Then, in response to the free space in memory 104 being at least equal to the memory usage amount "A [GB]", the information processing device 101 executes process 1 using the GPU 103b.

[0038] Thus, the information processing device 101 can suppress errors that occur when allocating the GPU 103 for the processing of the target program 110, such as when memory space required for the execution of the target program 110 cannot be secured, thereby improving the utilization rate of the GPU 103.

[0039] In the example shown in Figure 1, when allocating the GPU 103b for process 1 of the target program 110, there is insufficient free memory 104. Therefore, the information processing device 101 waits for the GPU 103b to execute process 1 of the target program 110 until the memory area in memory 104 allocated to another program is released. This allows the information processing device 101 to secure sufficient memory area in memory 104 on the GPU 103 when process 1 is executed, even if the timing of releasing the GPU 103 is set to the beginning of the program's execution, thereby suppressing errors such as out-of-memory errors.

[0040] (Example of system configuration for information processing system 200) Next, we will describe an example of the system configuration of the information processing system 200, which includes the information processing device 101 shown in Figure 1. Here, we will explain using the example of applying the information processing device 101 shown in Figure 1 to the execution control device 201 within the information processing system 200.

[0041] Figure 2 is an explanatory diagram showing an example of the system configuration of the information processing system 200. In Figure 2, the information processing system 200 includes an execution control unit 201 and a user terminal 202. In the information processing system 200, the execution control unit 201 and the user terminal 202 are connected via a wired or wireless network 210. The network 210 is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0042] Here, the execution control unit 201 is a computer that controls the execution of program processing and has a memory usage management table 220 and a free memory management table 230. The execution control unit 201 is, for example, a server. The contents of the memory usage management table 220 and the free memory management table 230 will be described later with reference to Figures 4 and 5.

[0043] User terminal 202 is a computer used by users of the information processing system 200. User terminal 202 can be, for example, a PC (Personal Computer), a tablet PC, or a smartphone.

[0044] The user terminal 202 can execute a user program in the execution control unit 201 by, for example, sending the user program to the execution control unit 201. The user program is the program to be executed and includes processes that are repeatedly executed using a machine learning model.

[0045] In this example, the execution control unit 201 is provided separately from the user terminal 202, but this is not the only option. For example, the execution control unit 201 may be implemented by the user terminal 202. Furthermore, the information processing system 200 may include multiple execution control units 201 and user terminals 202.

[0046] (Example hardware configuration of the execution control unit 201) Next, we will describe an example of the hardware configuration of the execution control unit 201.

[0047] Figure 3 is a block diagram showing an example of the hardware configuration of the execution control unit 201. In Figure 3, the execution control unit 201 includes a CPU 301, memory 302, GPU 303, GPU memory 304, communication interface 305, disk drive 306, disk 307, portable recording medium interface 308, and portable recording medium 309. Each component is connected by a bus 300.

[0048] Here, the CPU 301 is responsible for the overall control of the execution control unit 201. The GPU 303 performs computational processing such as image processing and natural language processing. The CPU 301 and GPU 303 may have multiple cores. The GPU 303 may include, for example, two or more GPUs (devices). The memory 302 may include, for example, ROM (Read Only Memory) or RAM (Random Access Memory). Programs stored in the memory 302 are loaded into the CPU 301, causing the CPU 301 to execute the coded processing. The GPU memory 304 is memory dedicated to the GPU 303. The GPU memory 304 may be, for example, VRAM (Video RAM).

[0049] The following explanation uses the example of GPU303 containing n GPUs (devices) (n: a natural number greater than or equal to 2). Furthermore, the n GPUs (devices) are sometimes denoted as "GPU#1~#n," and any GPU among GPU#1~#n is sometimes denoted as "GPU#i" (i=1,2,…,n). Additionally, the memory of GPU#i is sometimes denoted as "memory#i." Memory#i corresponds to the memory area occupied by GPU#i within GPU memory 304.

[0050] The communication interface 305 is connected to the network 210 via a communication line, and through the network 210, it is connected to an external computer (for example, the user terminal 202 shown in Figure 2). The communication interface 305 manages the interface between the network 210 and the inside of the device, and controls the input and output of data from the external computer. The communication interface 305 is, for example, a modem or a LAN adapter.

[0051] The disk drive 306 controls the reading and writing of data to the disk 307 according to the control of the CPU 301. The disk 307 stores the data written under the control of the disk drive 306. The disk 307 is, for example, a magnetic disk, an optical disk, etc.

[0052] The portable recording medium interface 308 controls the reading and writing of data to the portable recording medium 309 according to the control of the CPU 301. The portable recording medium 309 stores the data written under the control of the portable recording medium interface 308. The portable recording medium 309 is, for example, a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory.

[0053] In addition to the components described above, the execution control device 201 may also have, for example, an input device, a display, a printer, a scanner, a microphone, a speaker, etc. Furthermore, the execution control device 201 does not necessarily have to have, for example, the portable recording medium I / F 308 and the portable recording medium 309 among the components described above.

[0054] (Contents stored in memory usage management table 220 and free memory management table 230) Next, using Figure 4, the contents of the memory usage management table 220 and the free memory management table 230 of the execution control unit 201 will be described. The memory usage management table 220 and the free memory management table 230 are implemented by storage devices such as the memory 302 and disk 307 shown in Figure 3.

[0055] Figure 4 is an explanatory diagram showing an example of the contents stored in the memory usage management table 220. In Figure 4, the memory usage management table 220 has fields for process ID and memory usage, and by setting information in each field, it stores memory usage information (for example, memory usage information 400-1 to 400-3) as records.

[0056] Here, the process ID is an identifier that uniquely identifies a process. A process corresponds to a user program. Memory usage indicates the amount of memory used when the process (user program) is executed. Memory usage corresponds to the storage capacity of the memory area occupied by the process (user program). For example, memory usage information 400-1 shows the memory usage of process p1, "AA [GB]".

[0057] Figure 5 is an explanatory diagram showing an example of the contents stored in the free memory management table 230. In Figure 5, the free memory management table 230 has fields for GPUID and free capacity, and by setting information in each field, it stores free memory information (for example, free memory information 500-1 to 500-3) as records.

[0058] Here, GPUID is an identifier that uniquely identifies GPU#i. Free space indicates the free space in memory#i of GPU#i. Note that the free space in memory#i refers to the amount of unused storage area within memory#i. For example, free memory information 500-1 indicates the free space of memory#1 of GPU#1, "XX [GB]".

[0059] (Example of functional configuration of the execution control unit 201) Next, we will describe an example of the functional configuration of the execution control unit 201.

[0060] Figure 6 is a block diagram showing an example of the functional configuration of the execution control device 201. In Figure 6, the execution control device 201 includes an acquisition unit 601, an execution unit 602, a measurement unit 603, a data management unit 604, an allocation unit 605, a specific unit 606, a determination unit 607, an execution control unit 608, a memory release unit 609, and a storage unit 610. The acquisition unit 601 to the memory release unit 609 function as the control unit 600, and specifically, for example, it realizes its function by having the CPU 301 or GPU 303 execute a program stored in a storage device such as the memory 302, GPU memory 304, disk 307, or portable recording medium 309 shown in Figure 3, or by using the communication I / F 305. The processing results of each functional unit are stored in a storage device such as the memory 302 or disk 307.

[0061] The storage unit 610 is implemented by a storage device such as a memory 302 or a disk 307. Specifically, for example, the storage unit 610 stores the memory usage management table 220 shown in Figure 4 and the free memory management table 230 shown in Figure 5. Here, we describe the case where the storage unit 610 is included in the execution control unit 201, but it is not limited to this. For example, the storage unit 610 may be included in a computer different from the execution control unit 201 and accessible from the execution control unit 201 via the network 210.

[0062] The acquisition unit 601 acquires the target program. The target program is a program to be executed, such as a user program. Specifically, for example, the acquisition unit 601 acquires the target program by receiving it from the user terminal 202 shown in Figure 2. Alternatively, the target program may be acquired through user input using an input device (not shown).

[0063] In the following explanation, the target program may be referred to as "User Program P." User Program P includes processes that are repeatedly executed using a machine learning model. The target program 110 shown in Figure 1 corresponds to User Program P, for example.

[0064] The execution unit 602 executes the processing of the acquired user program P in accordance with the control of the execution control unit 608. Specifically, for example, when the execution unit 602 executes the processing of user program P, it makes a GPU request to the allocation unit 605. Here, the GPU request is a request to allocate GPU#i for the processing of user program P.

[0065] The execution unit 602 then executes the processing of the user program P using the CPU 301 or GPU#i identified from the allocation result from the execution control unit 608. A GPU request is made, for example, for each process that is repeatedly executed in the user program P. The GPU request may include, for example, a process ID. The GPU request may also include memory usage.

[0066] The measurement unit 603 measures the memory usage of the user program P. The memory usage corresponds to the amount of memory space occupied by the execution of the user program P. For example, if the user program P is executed by the CPU 301, the memory usage corresponds to the amount of memory space occupied by the user program P within the memory 302.

[0067] Furthermore, when user program P is executed on GPU#i, the memory usage corresponds to the amount of memory occupied by user program P within the memory #i of GPU#i. Specifically, for example, the measurement unit 603 may use a performance counter to measure the memory usage of user program P.

[0068] Furthermore, the measurement unit 603 may measure memory usage only during the first execution of the user program P. Alternatively, the measurement unit 603 may measure memory usage each time the user program P is executed. Alternatively, the measurement unit 603 may measure memory usage during a predetermined number of executions of the user program P (for example, during the first to fifth executions).

[0069] The data management unit 604 manages the memory usage of user program P. Specifically, for example, it records the memory usage measured by the measurement unit 603 in the memory usage management table 220 shown in Figure 4, in association with the process ID of user program P.

[0070] Furthermore, memory usage may be measured each time user program P is executed, or after a predetermined number of executions. In this case, the data management unit 604 may calculate the average of the measured memory usage and record the calculated average memory usage in the memory usage management table 220, associating it with the process ID of user program P. Alternatively, the data management unit 604 may identify the maximum memory usage among the measured memory usage and record the identified maximum memory usage in the memory usage management table 220, associating it with the process ID of user program P.

[0071] Furthermore, the processing content of user program P may be such that there is no significant difference in memory usage between processes that are executed repeatedly. In this case, the data management unit 604 may record only the memory usage measured during the first execution in the memory usage management table 220.

[0072] The allocation unit 605 allocates resources for the processing of user program P. The resources are, for example, a CPU 301 and a GPU 303 that can be shared by two or more programs. Specifically, for example, the allocation unit 605 performs scheduling processing to allocate CPU 301 or GPU#i for the processing of user program P in response to a GPU request. The processing of user program P is identified, for example, by the process ID included in the GPU request.

[0073] To explain in more detail, for example, the allocation unit 605 measures the performance values ​​when the processing of user program P is executed on both CPU 301 and GPU#i. Then, based on the measured performance values, the allocation unit 605 decides whether to assign CPU 301 or GPU#i to the processing of user program P.

[0074] For example, the allocation unit 605 may determine that the priority is high and allocate GPU#i to the processing of user program P if the performance value when executed on GPU#i is above a threshold and above the performance value when executed on CPU301. On the other hand, the allocation unit 605 may determine that the priority is low and allocate CPU301 to the processing of user program P if the performance value when executed on GPU#i is not above a threshold and above the performance value when executed on CPU301.

[0075] Any existing technology may be used for the scheduling process to allocate CPU 301 or GPU#i. However, when allocating CPU 301 or GPU#i, the allocation unit 605 does not need to consider the available memory capacity of memory 302 and GPU#i's memory #i.

[0076] The identification unit 606 identifies the available memory #i of GPU #i when GPU #i is assigned to process user program P. The available memory #i is the amount of memory within memory #i that is not allocated to any program. Specifically, for example, the identification unit 606 may identify the available memory #i of GPU #i by querying a management function such as a task manager.

[0077] To explain in more detail, for example, the identification unit 606 may identify the available memory capacity of memory #i for each GPU #i included in GPU #1 to #n at predetermined time intervals (for example, intervals of several seconds to tens of seconds). The identified available memory capacity of memory #i is stored in the free memory management table 230 shown in Figure 5, for example, in association with the GPUID of GPU #i. Then, when GPU #i is assigned to the processing of user program P, the identification unit 606 may refer to the free memory management table 230 to identify the available memory capacity of memory #i of GPU #i.

[0078] The determination unit 607 compares the free memory capacity of the identified GPU#i with the memory usage measured during the previous execution of the user program P. Then, the information processing device 101 determines, based on the comparison result, whether or not there is insufficient free memory capacity of GPU#i.

[0079] Specifically, for example, the determination unit 607 refers to the memory usage management table 220 to identify the memory usage corresponding to the process ID of the user program P. Then, the determination unit 607 compares the free space of the identified memory #i with the identified memory usage. Here, if the free space of memory #i is greater than or equal to the memory usage, it may be determined that there is no shortage of free space in memory #i. On the other hand, if the free space of memory #i is less than the memory usage, it may be determined that there is a shortage of free space in memory #i.

[0080] Furthermore, the determination unit 607 may determine that there is no shortage of free memory in memory #i if the value obtained by subtracting the memory usage from the free capacity of memory #i is greater than or equal to a predetermined value. On the other hand, if the value obtained by subtracting the memory usage from the free capacity of memory #i is less than the predetermined value, the determination unit 607 may determine that there is a shortage of free memory in memory #i. The predetermined value is a value greater than 0 and can be set arbitrarily.

[0081] Note that the memory usage of user program P is not measured during the first execution of its processing. In this case, the determination unit 607 may determine that there is sufficient free memory in memory #i, or it may determine that there is sufficient free memory in memory #i.

[0082] The execution control unit 608 executes the processing of user program P using GPU#i if there is sufficient free memory #i on GPU#i. If there is insufficient free memory #i on GPU#i, the execution control unit 608 waits for the GPU#i to release memory #i allocated to another program before executing the processing of user program P.

[0083] Specifically, for example, if there is sufficient free memory #i on GPU#i, the execution control unit 608 notifies the execution unit 602 of the allocation result indicating that GPU#i has been allocated as a response to the GPU request. As a result, the execution unit 602 executes the processing of the user program P using GPU#i, which is identified from the allocation result from the execution control unit 608.

[0084] In this case, the assigned GPU#i may be the same as the previous GPU#j that was assigned during the previous execution of the user program P (i=j). In this case, the execution unit 602 uses the model data representing the machine learning model stored in memory #i of GPU#i to execute the processing of the user program P using GPU#i.

[0085] Furthermore, if there is insufficient free memory in memory #i of GPU #i, the execution control unit 608 waits for the GPU #i to execute the user program P until, for example, a storage area within memory #i is freed and the free capacity of memory #i becomes at least equal to or greater than the specified memory usage.

[0086] Then, upon completion of the waiting period, the execution control unit 608 notifies the execution unit 602 of the allocation result, indicating that GPU#i has been allocated as a response to the GPU request. As a result, the execution unit 602 executes the processing of the user program P using GPU#i, which is identified from the allocation result from the execution control unit 608.

[0087] The execution control unit 608 may terminate the wait if the memory area in memory #i that has been allocated to at least one other program is released. Even in this case, it is expected that the occurrence of errors such as out-of-memory errors will be reduced compared to when the processing of user program P is executed immediately when GPU #i is allocated.

[0088] Furthermore, if the execution control unit 608 is allocated CPU 301 for processing user program P, it notifies the execution unit 602 of the allocation result indicating that CPU 301 has been allocated as a response to the GPU request. As a result, the execution unit 602 executes the processing of user program P using CPU 301, which is identified from the allocation result from the execution control unit 608.

[0089] Furthermore, the execution control unit 201 may choose not to perform the processing of the identification unit 606 and the determination unit 607 during the first execution of the user program P, since the memory usage of the user program P has not yet been measured. In this case, the allocation unit 605 may allocate the CPU 301 to the processing of the user program P during the first execution of the user program P. This prevents the execution control unit 201 from allocating GPU#i to the processing of the user program P even though there is insufficient free memory #i on GPU#i.

[0090] The memory deallocation unit 609 determines whether GPU#i, when assigned to the processing of user program P, is the same as the previous GPU#j assigned during the previous execution of user program P. Here, information identifying the previous GPU (e.g., GPUID) is stored in memory 302 in association with information identifying the process (user program P) (e.g., process ID).

[0091] Specifically, for example, the memory release unit 609 determines that the GPU#i assigned this time is the same as the GPU#j assigned last time if the GPUID of the GPU#i assigned this time is the same as the GPUID of the GPU#j assigned last time (i=j). Also, the memory release unit 609 determines that the GPU#i assigned this time is different from the GPU#j assigned last time if the GPUID of the GPU#i assigned this time is different from the GPU#j assigned last time (i≠j).

[0092] Note that the processing of the previous user program P may have been executed on CPU 301. In this case, the memory deallocation unit 609 does not need to determine whether it is the same as the previous GPU#j.

[0093] Then, if the memory release unit 609 is different from the previous GPU#j, it releases the memory area allocated to user program P in memory#j of the previous GPU#j. The memory area allocated to user program P is the memory area occupied by user program P. Releasing the memory area allocated to user program P in memory#j (memory release) may be done, for example, by executing a memory release command.

[0094] Specifically, for example, the memory release unit 609 releases the memory area allocated to user program P that contains the model data stored in memory #j of GPU #j from the previous session. The model data is information representing the machine learning model used by user program P's processing, and includes, for example, information identifying the machine learning model's algorithm and parameters (weight parameters).

[0095] To explain in more detail, for example, if the memory release unit 609 is different from the previous GPU#j, it transfers the model data from memory#j of the previous GPU#j to memory#i of GPU#i. Then, the memory release unit 609 releases the memory area in memory#j of the previous GPU#j where the model data was stored.

[0096] The memory release unit 609 can shorten the model transfer time by directly transferring model data between memory #i and #j of GPU #i and #j. However, the memory release unit 609 may also transfer the model data from memory #j of GPU #j to memory #i of GPU #i via CPU 301 or memory 302.

[0097] Furthermore, when the CPU 301 is assigned to process the user program P, the memory release unit 609 may determine whether or not GPU#j was assigned during the previous execution of the user program P. If GPU#j was assigned during the previous execution, the memory release unit 609 releases the memory area within memory#j of GPU#j that was previously allocated to the user program P.

[0098] Specifically, for example, the memory release unit 609 releases the memory area in memory #j of the previous GPU #j that was allocated to the user program P, where the model data was stored. More specifically, for example, the memory release unit 609 transfers the model data from memory #j of the previous GPU #j to memory 302 of the CPU 301. Then, the memory release unit 609 releases the memory area in memory #j of the previous GPU #j that was stored.

[0099] In this way, the memory deallocation unit 609 sets the timing of memory deallocation for the GPU (the previous GPU#j) to the start of execution of the user program P. As a result, when the execution control unit 201 is assigned the same GPU#i (i=j) as in the previous execution, it can reuse the model data in memory#i on GPU#i and omit model transfer.

[0100] However, among the data required to execute user program P, data other than the model data representing the machine learning model (input / output data) is often different for each execution of user program P. Therefore, even if the same GPU is allocated as in the previous execution, it cannot be expected that the previous input / output data will be reused.

[0101] Therefore, when the processing of user program P by GPU#i is completed, the memory release unit 609 may release the memory area in memory#i of GPU#i that contains the input / output data, which was allocated to the processing of user program P. This allows the execution control unit 201 to secure free space for the input / output data in memory#i of GPU#i, preventing the situation where memory area is unnecessarily occupied.

[0102] The functional units of the execution control unit 201 (acquisition unit 601 to memory release unit 609) may be implemented by multiple computers within the information processing system 200 (for example, the execution control unit 201, user terminal 202, execution devices not shown, etc.). In this case, communication between functional units of different computers is performed, for example, by sending and receiving data between functional units via the network 210. For example, the execution unit 602, measurement unit 603, and memory release unit 609 may be implemented by execution devices (not shown) different from the execution control unit 201. In this case, the execution device (not shown) has arithmetic units equivalent to a CPU 301 and GPU 303 (GPU#1 to #n) used to execute the processing of the user program P.

[0103] (Example of operation of the execution control unit 201) Next, an example of the operation of the execution control device 201 will be explained using Figure 7.

[0104] Figure 7 is an explanatory diagram showing an example of the operation of the execution control unit 201. In Figure 7, the execution unit 602, the measurement unit 603, and the memory release unit 609 are assumed to be implemented by the deep learning framework 701 of the execution control unit 201. The deep learning framework 701 provides the necessary functions for performing processes related to deep learning (learning process, estimation process). The deep learning framework 701 is called, for example, by a user program P via an API (Application Programming Interface).

[0105] Furthermore, the data management unit 604, the allocation unit 605, the identification unit 606, the determination unit 607, and the execution control unit 608 are implemented by the GPU Assigner 702. The GPU Assigner 702 is a function that manages the allocation of GPUs 303 (GPU #1 to #n).

[0106] The deep learning framework 701, using the execution unit 602, makes a GPU request to the allocation unit 605 when executing the processing of the user program P. At this time, the deep learning framework 701, using the measurement unit 603, sends the memory usage (measurement result) measured during the execution of the user program P to the data management unit 604. Note that the memory usage (measurement result) may be included in the GPU request.

[0107] The GPU Assigner 702, via the data management unit 604, records the memory usage (measurement result) received from the deep learning framework 701 in the memory usage management table 220, associating it with the process ID of the user program P.

[0108] Furthermore, the GPU Assigner 702, via the allocation unit 605, performs scheduling processing to assign either CPU 301 or GPU#i to the processing of user program P in response to a GPU request. Here, let's assume that GPU#1 is assigned to the processing of user program P.

[0109] In this case, the GPU Assigner 702 uses the identification unit 606 to refer to the free memory management table 230 and identify the free capacity "XX [GB]" of memory #1 of GPU #1. Next, the GPU Assigner 702 uses the determination unit 607 to refer to the memory usage management table 220 and identify the memory usage corresponding to the process ID of user program P.

[0110] Here, we assume the process ID of user program P is "p1". In this case, the memory usage "AA [GB]" corresponding to the process ID "p1" of user program P is identified. The GPU Assigner 702 uses the determination unit 607 to compare the identified free memory capacity "XX [GB]" of memory #1 of GPU #1 with the identified memory usage "AA [GB]". Then, the GPU Assigner 702 uses the determination unit 607 to determine whether or not there is insufficient free memory in memory #1 of GPU #1 based on the comparison result.

[0111] Here, using Figure 8, we will explain an example of how to determine whether or not there is insufficient free memory in memory #1 of GPU #1.

[0112] Figure 8 is an explanatory diagram illustrating an example of how to determine whether or not there is insufficient free memory. In Figure 8, the memory usage 801 is the memory usage measured during the first execution of user program P, and corresponds to the identified memory usage "AA [GB]".

[0113] In example (8-1), the free space in memory #1 is set to "free space 802". Free space 802 corresponds to the specified free space "XX [GB]". In this case, the GPU Assigner 702 compares the memory usage 801 with the free space 802 of memory #1 using the determination unit 607. Here, although a portion of memory #1 is being used by another process, the free space 802 of memory #1 is greater than or equal to the memory usage 801 (XX ≥ AA). In this case, the GPU Assigner 702 determines using the determination unit 607 that there is no shortage of free space in memory #1.

[0114] In the example of (8 - 2), the free capacity of Memory #1 is set as "Free Capacity 803". The Free Capacity 803 corresponds to the specified free capacity "XX [GB]". In this case, the GPU Assigner 702 compares the memory usage 801 with the free capacity 803 of Memory #1 through the determination unit 607. Here, most of Memory #1 is being used by another process, and the free capacity 803 of Memory #1 is less than the memory usage 801 (XX < AA). In this case, the GPU Assigner 702 determines through the determination unit 607 that there is insufficient free space in Memory #1.

[0115] Returning to the description of FIG. 7, when there is no shortage of free space in Memory #1 of GPU #1, the GPU Assigner 702 notifies the execution unit 602, through the execution control unit 608, of an allocation result indicating that GPU #1 has been allocated as a response to the GPU request.

[0116] Also, when there is a shortage of free space in Memory #1 of GPU #1, the GPU Assigner 702 waits, through the execution control unit 608, for the execution of the user program P by GPU #1 until the storage area in Memory #1 allocated to another program is released. Then, in response to the end of the waiting, the GPU Assigner 702 notifies the execution unit 602, through the execution control unit 608, of an allocation result indicating that GPU #1 has been allocated as a response to the GPU request.

[0117] Thereby, the GPU Assigner 702 can prevent errors such as out - of - memory from occurring when it is unable to secure a storage area for storing the data required for the execution of the user program P.

[0118] The deep learning framework 701 determines, through the memory release unit 609, whether the allocated GPU #1 is the same as the previous GPU #j allocated during the previous execution of the user program P. Here, it is assumed that it is the same as the previous GPU #j (j = 1).

[0119] In this case, the deep learning framework 701 does not release the memory area allocated to the user program P in memory #1 of GPU#1 (the previous GPU#j). Then, the deep learning framework 701, via the execution unit 602, executes the processing of the user program P on GPU#1, which is identified from the allocation result.

[0120] This allows the deep learning framework 701 to reuse the model data located in memory #1 on GPU #1, thus eliminating the need for model transfer.

[0121] Furthermore, if the assigned GPU#1 is different from the previous GPU#j, the deep learning framework 701 uses the memory deallocation unit 609 to free the memory area allocated to the user program P within memory#j of the previous GPU#j.

[0122] Specifically, for example, the deep learning framework 701, using the memory release unit 609, releases the memory area where the model data in memory #j of GPU #j was stored, which was allocated to the user program P. Then, the deep learning framework 701, using the execution unit 602, executes the processing of the user program P on GPU #1, which is identified from the allocation result.

[0123] This allows the deep learning framework 701 to release the memory area containing the model data in memory #j, which is no longer used in this process, because the allocated GPU has changed, thus preventing unnecessary memory occupancy.

[0124] (Example of data transfer before and after applying this memory release process) Next, we will explain data transfer examples before and after applying this memory release process using Figures 9 and 10.

[0125] Figures 9 and 10 are explanatory diagrams showing data transfer examples before and after the application of this memory release processing method. Figure 9 shows an example of data transfer before the application of this memory release processing method.

[0126] First, when executing Epoch 0 of user program 1, the GPU is allocated to Epoch 0. In this case, input / output data transfer and model transfer are performed to transfer input / output data and model data to the GPU's memory, and GPU processing (data processing by the GPU) is performed. Then, when the GPU processing is finished, model transfer and input / output data transfer are performed to move the model data and input / output data out of the GPU's memory.

[0127] Next, when executing Epoch 1 of user program 1, a GPU is allocated for Epoch 1. In this case, input / output data transfer and model transfer are performed to transfer model data and input / output data to the GPU's memory, and GPU processing is carried out. Then, when the GPU processing is completed, model transfer and input / output data transfer are performed to move the model data and input / output data out of the GPU's memory.

[0128] Furthermore, if the CPU is assigned to an Epoch of User Program 1 (for example, Epochs 2-4), the CPU processing will be executed without input / output data transfer or model transfer, because the input / output data and model data are already in the CPU's memory. The same applies to User Program 2 as to User Program 1.

[0129] Thus, before applying this memory release method, when a GPU is allocated, each time user programs 1 and 2 complete their processing cycle, model transfer and input / output data transfer are performed to free up the GPU and release memory.

[0130] Figure 10 shows an example of data transfer after applying this memory release processing method.

[0131] First, when executing Epoch 0 of user program 1, the GPU is allocated to Epoch 0. In this case, input / output data transfer and model transfer are performed to transfer input / output data and model data to the GPU's memory, and GPU processing (data processing by the GPU) is performed. Then, when the GPU processing is finished, input / output data transfer is performed to move the input / output data out of the GPU's memory. At this point, model transfer to move the model data is not performed.

[0132] Next, when executing Epoch 1 of user program 1, the same GPU as the previous time is assigned to Epoch 1. In this case, input / output data transfer is performed to transfer input / output data to the GPU's memory, and GPU processing is performed. Model data is not transferred because it can be reused. Then, when the GPU processing is finished, input / output data transfer is performed to move the input / output data from the GPU's memory. At this point, model transfer is not performed to move the model data.

[0133] Next, when executing Epoch 2 of user program 1, the same GPU as the previous one is assigned to Epoch 2. In this case, input / output data transfer is performed to transfer input / output data to the GPU's memory, and GPU processing is performed. Model data is not transferred because it can be reused. Then, when the GPU processing is finished, input / output data transfer is performed to move the input / output data from the GPU's memory. At this point, model transfer is not performed to move the model data.

[0134] Next, when executing Epoch 3 of user program 1, the CPU is allocated to Epoch 3. In this case, during the execution of Epoch 2, a model transfer is performed to move the model data from the GPU memory, and CPU processing is carried out.

[0135] Subsequently, when the CPU is assigned to Epochs 4 and 5 of user program 1, the CPU processing is executed without input / output data transfer or model transfer because the input / output data and model data are already in the CPU's memory.

[0136] Thus, after applying this memory release method, by timing the GPU memory release to the beginning of the execution of user program 1's process (Epoch), if the same GPU as the previous execution is allocated, the model data in the GPU's memory can be reused, and model transfer can be omitted. As a result, after applying this memory release method, it becomes possible to execute Epochs 0 and 2 in approximately the same amount of time as when Epochs 0 and 1 were executed before the application of this memory release method.

[0137] The same applies to User Program 2 as to User Program 1. However, if the GPU is allocated for Epoch 2 of User Program 2 and there is insufficient memory available on that GPU, the execution of Epoch 2 will be put into a waiting state until another program (for example, User Program 1) frees up memory.

[0138] (Various processing procedures of the execution control unit 201) Next, the various processing procedures of the execution control unit 201 will be described. First, the memory release processing procedure of the execution control unit 201 will be described using Figures 11 and 12. The memory release processing of the execution control unit 201 is performed, for example, by the deep learning framework 701 (execution unit 602, measurement unit 603, and memory release unit 609) shown in Figure 7.

[0139] Figures 11 and 12 are flowcharts illustrating an example of the memory release process procedure of the execution control unit 201. In the flowchart of Figure 11, first, the execution control unit 201 sends a GPU request to the GPU Assigner 702 (allocation unit 605) when executing the processing of user program P (step S1101). The GPU request includes, for example, the process ID and memory usage of user program P.

[0140] Next, the execution control unit 201 determines whether or not GPU#i has been assigned to the processing of user program P (step S1102). Specifically, for example, if the execution control unit 201 receives an assignment result indicating that GPU#i has been assigned, it determines that GPU#i has been assigned to the processing of user program P. Also, if the execution control unit 201 receives an assignment result indicating that CPU301 has been assigned, it determines that GPU#i has not been assigned to the processing of user program P.

[0141] If GPU#i is assigned here (step S1102: Yes), the execution control unit 201 determines whether the GPUID of GPU#i has switched from the previous GPU (step S1103). The previous GPU is GPU#j, which was assigned during the previous execution of user program P.

[0142] If the GPUID changes at this point (step S1103: Yes), the execution control unit 201 copies the model data from memory #j of the previous GPU #j to memory 302 of the CPU 301 (step S1104). Then, the execution control unit 201 releases the memory area where the model data in memory #j on the previous GPU #j was stored (step S1105), and proceeds to step S1107.

[0143] In step S1104, the execution control unit 201 may directly transfer model data between the memories #i and #j of the GPUs #i and #j. In this case, the execution control unit 201 can omit the processing in step S1107.

[0144] Furthermore, if the GPUID has not been switched in step S1103 (step S1103: No), the execution control unit 201 determines whether or not model data exists in memory #i on the allocated GPU #i (step S1106). If model data exists (step S1106: Yes), the execution control unit 201 proceeds to step S1108.

[0145] On the other hand, if model data is not available (step S1106: No), the execution control unit 201 copies the model data from the CPU 301's memory 302 to memory #i on GPU #i (step S1107). Then, the execution control unit 201 performs the execution process (GPU) of the user program P (step S1108). The execution process (GPU) of the user program P is the execution of the user program P by GPU #i.

[0146] Next, the execution control unit 201 releases the memory area in memory #i on GPU #i where the input / output data is stored (step S1109). Then, the execution control unit 201 determines whether or not there are any unexecuted processes in the user program P (step S1110). If there are unexecuted processes (step S1110: Yes), the execution control unit 201 returns to step S1101.

[0147] On the other hand, if there are no unexecuted processes (step S1110: No), the execution control unit 201 releases the memory area in memory #i on the allocated GPU #i where the model data is stored (step S1111), and terminates the series of processes according to this flowchart. However, if there is no model data in memory #i on the GPU #i, the execution control unit 201 skips the process in step S1111.

[0148] Furthermore, if GPU#i is not assigned in step S1102 (step S1102: No), the execution control unit 201 proceeds to step S1201 shown in Figure 12.

[0149] In the flowchart of Figure 12, first, the execution control unit 201 determines whether or not CPU 301 was allocated during the previous execution of user program P (step S1201). In other words, the execution control unit 201 determines whether or not GPU#j was allocated during the previous execution of user program P.

[0150] If CPU 301 has been assigned at this point (step S1201: Yes), the execution control unit 201 proceeds to step S1204.

[0151] On the other hand, if CPU 301 is not allocated (step S1201: No), the execution control unit 201 copies the model data from memory #j on the previous GPU #j to memory 302 on CPU 301 (step S1202). Next, the execution control unit 201 releases the memory area where the model data in memory #j on the previous GPU #j was stored (step S1203).

[0152] Then, the execution control unit 201 performs the execution process (CPU) of the user program P (step S1204), and after completion, proceeds to step S1110 shown in Figure 11. The execution process (CPU) of the user program P is the execution of the user program P by the CPU 301.

[0153] As a result, when the execution control unit 201 repeatedly executes the processing of the user program P, it can improve the utilization efficiency of GPU#i by setting the timing of freeing the allocated GPU#i to the beginning of the execution of the processing of the user program P.

[0154] In step S1103, if the GPUID is switched (step S1103: Yes), and the CPU 301 was assigned during the previous execution of user program P, the execution control unit 201 may skip steps S1104 and S1105.

[0155] Furthermore, the execution control unit 201 measures the memory usage of the user program P's processing during its initial execution (first execution). The execution control unit 201 may then include the measured memory usage (measurement result) in the GPU request when executing subsequent processes of the user program P.

[0156] Next, the execution control processing procedure of the execution control device 201 will be explained using Figure 13. The execution control processing of the execution control device 201 is performed, for example, by the GPU Assigner 702 (data management unit 604, allocation unit 605, identification unit 606, determination unit 607, and execution control unit 608) shown in Figure 7.

[0157] Figure 13 is a flowchart showing an example of the execution control processing procedure of the execution control unit 201. In the flowchart of Figure 13, first, the execution control unit 201 determines whether or not it has received a GPU request from the deep learning framework 701 (execution unit 602) (step S1301). Here, the execution control unit 201 waits to receive a GPU request (step S1301: No).

[0158] Then, when the execution control unit 201 receives a GPU request (step S1301: Yes), it performs scheduling processing to assign either CPU 301 or GPU#i to the processing of user program P (step S1302). The processing of user program P is identified, for example, by the process ID included in the GPU request.

[0159] Furthermore, if the execution control unit 201 includes memory usage in the GPU request, for example, it records the memory usage included in the GPU request in the memory usage management table 220, associating it with the process ID of the user program P.

[0160] Next, the execution control unit 201 determines whether or not it has allocated GPU#i for processing the user program P (step S1303). If it has allocated CPU301 (step S1303: No), the execution control unit 201 sends the allocation result indicating that CPU301 has been allocated to the deep learning framework 701 (step S1304), and terminates the series of processes according to this flowchart.

[0161] On the other hand, if GPU#i is allocated (step S1303: Yes), the execution control unit 201 identifies the available memory #i of GPU#i (step S1305). Next, the execution control unit 201 refers to the memory usage management table 220 to identify the memory usage corresponding to the process ID of user program P (step S1306).

[0162] Then, the execution control unit 201 determines whether or not there is insufficient free memory for GPU#i based on the result of comparing the free memory for GPU#i with the measured memory usage (step S1307).

[0163] If memory #i is insufficient (step S1307: Yes), the execution control unit 201 waits for the GPU #i to process the user program P until the memory area in memory #i allocated to another program is freed (step S1308), and then proceeds to step S1305.

[0164] Furthermore, in step S1307, if there is no shortage of free memory #i on GPU#i (step S1307: No), the execution control unit 201 sends an allocation result indicating that GPU#i has been allocated to the deep learning framework 701 (step S1309), and terminates the series of processes according to this flowchart.

[0165] This allows the execution control unit 201 to prevent errors caused by insufficient memory on GPU#i when allocating GPU#i for processing the user program P.

[0166] (Example of changes in GPU usage) Next, we will explain an example of changes in GPU usage using Figure 14.

[0167] Figure 14 is an explanatory diagram showing an example of changes in GPU usage. Figure 14 shows the processing image when the processing (Epoch) of user program P is repeatedly executed on the same GPU. Processing image 1401 shows the processing on the GPU (Epoch 0 to 3) before the application of this memory release processing method.

[0168] Processing image 1402 shows the processing on the GPU (Epoch 0-5) after applying this memory release method. In processing image 1402, due to the reduction in model transfer time, more processing can be performed in the same amount of time as in processing image 1401, and the GPU utilization has improved. Processing image 1402 shows that the more consecutive executions there are on the same GPU, the higher the processing efficiency becomes.

[0169] As described above, according to the execution control device 201 of this embodiment, when GPU#i is assigned to process user program P, the available memory #i of GPU#i can be identified. User program P includes processing that is repeatedly executed using a machine learning model. GPU#i is one of the GPUs 303 (GPU#1 to #n) that can be shared by two or more programs. Furthermore, according to the execution control device 201, it is possible to determine whether or not there is insufficient free memory #i of GPU#i based on the result of comparing the identified free memory #i of GPU#i with the memory usage measured during a previous execution of user program P's processing. Specifically, for example, the execution control device 201 determines that there is no shortage of free memory #i if the free memory #i is greater than or equal to the memory usage. On the other hand, the execution control device 201 determines that there is insufficient free memory #i if the free memory #i is less than the memory usage. According to the execution control unit 201, if there is sufficient free memory #i on GPU#i, the user program P can be executed by GPU#i. Also, according to the execution control unit 201, if there is insufficient free memory #i on GPU#i, the execution of the user program P by GPU#i can be delayed until the memory area in memory #i allocated to another program is released.

[0170] This allows the execution control unit 201 to suppress errors that occur when allocating GPU#i for the processing of user program P, due to inability to secure the memory area required for the execution of user program P, thereby improving the utilization rate of GPU#i.

[0171] Furthermore, according to the execution control unit 201, when GPU#i is assigned to the processing of user program P, it can determine whether GPU#i is the same as the previous GPU#j that was assigned during the previous execution of user program P. According to the execution control unit 201, if it is different from the previous GPU#j, it can release the memory area in memory#j of the previous GPU#j that was assigned to user program P. Specifically, for example, the execution control unit 201 releases the memory area in memory#j of the previous GPU#j that contains model data representing the machine learning model, which was assigned to user program P.

[0172] As a result, when the execution control unit 201 repeatedly executes the processing of user program P, it can set the timing of memory release for GPU#i to the beginning of the execution of user program P. Therefore, when the execution control unit 201 uses the same device (GPU#i) continuously, it is possible to reuse model data, reduce the number of model transfers, and improve the utilization efficiency of GPU#i.

[0173] Furthermore, according to the execution control unit 201, if it is the same as the previous GPU#j, the execution of the user program P by GPU#i can be performed using the model data representing the machine learning model stored in memory #i of GPU#i.

[0174] This allows the execution control unit 201 to reuse the model data located in memory #i on GPU#i, reducing the number of model transfers that can become a bottleneck when using large-scale machine learning models, and improving the utilization efficiency of GPU#i.

[0175] Furthermore, according to the execution control unit 201, if there is insufficient free memory #i on GPU#i, the execution of user program P by GPU#i can wait until memory #i allocated to other programs is released and the free capacity of memory #i becomes at least equal to or greater than the memory usage.

[0176] This allows the execution control unit 201 to wait for the execution of user program P until it has secured the memory capacity used during the previous execution of user program P, thereby preventing errors such as out-of-memory errors from occurring.

[0177] Furthermore, according to the execution control unit 201, when the processing of user program P by GPU#i is completed, the memory area within memory#i of GPU#i that was allocated for the processing of user program P, specifically the memory area where input / output data is stored, can be released. Input / output data is data required for the execution of user program P, excluding model data representing the machine learning model.

[0178] This allows the execution control unit 201 to reserve free space within memory #i of GPU #i for the input / output data, preventing the unnecessary occupancy of memory #i.

[0179] Furthermore, according to the execution control unit 201, when the CPU 301 is assigned to process the user program P from among the CPU 301 and GPU 303 (GPU #1 to #n), which can be shared by two or more programs, it can determine whether GPU #j was assigned during the previous execution of the user program P. According to the execution control unit 201, if GPU #j was assigned during the previous execution, it can release the memory area allocated to the user program P within memory #j of GPU #j that was allocated during the previous execution. Specifically, for example, the execution control unit 201 releases the memory area in memory #j of GPU #j that was allocated to the user program P and contains model data representing the machine learning model from the previous execution.

[0180] As a result, when the CPU 301 is assigned to process the user program P, the execution control unit 201 can free the memory of GPU#j that was allocated during the previous execution.

[0181] Based on these findings, the execution control device 201 according to this embodiment can reduce the number of model transfers and improve the utilization of GPU#i when repeatedly executing the processing of user program P using a CPU 301 and GPU 303 that can be shared by two or more programs, thereby enabling high-speed execution of user program P.

[0182] For example, even in a system where multiple users share resources, the execution control unit 201 can efficiently execute repetitive processing of user program P by switching between the CPU 301 and the GPU 303. Therefore, the execution control unit 201 enables high-speed training of machine learning models in application development such as AI (Artificial Intelligence) and advanced image recognition that utilize the GPU 303.

[0183] In the above explanation, the CPU 301 and GPU 303 were used as examples of the computing units that execute the program, but this is not limited to them. For example, a computing unit such as an NPU (Neural Network Processing Unit) may be used instead of the GPU 303, or a computing unit such as an NPU may be used in conjunction with the CPU 301 and GPU 303.

[0184] The memory release method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This memory release program is recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, DVD, or USB memory, and is executed when read from the recording medium by the computer. Furthermore, this memory release program may be distributed via a network such as the Internet.

[0185] Furthermore, the information processing device 101 (execution control device 201) described in this embodiment can also be implemented using application-specific ICs such as standard cells and structured ASICs (Application Specific Integrated Circuits), or PLDs (Programmable Logic Devices) such as FPGAs.

[0186] With regard to the embodiments described above, the following additional information is disclosed.

[0187] (Note 1) When a GPU shared by two or more programs is allocated to the processing of the target program, which is repeatedly executed using a machine learning model, the available memory capacity of one of the GPUs is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program will be executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by one of the GPUs will wait until a storage area in the memory allocated to another program is released. A memory release processing program characterized by having a computer perform the processing.

[0188] (Note 2) When any of the above GPUs are assigned to process the target program, it is determined whether or not that GPU is the same as the GPU that was assigned to the previous execution of the target program. If it is different from the previous GPU, the memory area allocated to the target program in the memory of the previous GPU is released. The memory release processing program according to Appendix 1, characterized in that it causes the computer to perform the processing.

[0189] (Note 3) The above release process is: The memory release processing program according to Appendix 2, characterized by releasing the memory area allocated to the target program, which stores model data representing the machine learning model in the GPU memory from the previous session.

[0190] (Note 4) The process that performs the above execution is: The memory release processing program according to Appendix 2 or 3, characterized in that, if it is the same as the previous GPU, it uses model data representing the machine learning model stored in the memory of one of the GPUs to execute the processing of the target program by one of the GPUs.

[0191] (Appendix 5) The process for making the above determination is, If the available memory capacity is greater than or equal to the amount of memory being used, it is determined that there is no shortage of available memory. If the available memory capacity is less than the amount of memory being used, it is determined that there is insufficient available memory. A memory deallocation program characterized by any one of the appendices 1 to 4.

[0192] (Appendix 6) The above waiting process is, A memory release processing program according to any one of the appendices 1 to 5, characterized in that it waits for the execution of processing of the target program by any of the GPUs until the memory area allocated to the other program is released and the free memory capacity becomes at least equal to or greater than the memory usage.

[0193] (Note 7) When processing of the target program by any of the aforementioned GPUs is completed, the memory area of ​​any of the aforementioned GPUs allocated to the target program that contains data other than the model data representing the machine learning model is released. A memory release processing program according to any one of the appendices 1 to 6, characterized in that it causes the computer to perform the processing.

[0194] (Note 8) Of the CPUs and GPUs that can be shared by two or more programs, when a CPU is assigned to the processing of the target program, it is determined whether or not a GPU was assigned during the previous execution of the processing of the target program. If a GPU was allocated during the previous execution, the memory area allocated to the target program within the GPU's memory allocated during the previous execution is released. A memory release processing program according to any one of the appendices 1 to 7, characterized in that it causes the computer to perform the processing.

[0195] (Note 9) When a GPU shared by two or more programs is allocated to the processing of the target program, which is repeatedly executed using a machine learning model, the available memory capacity of the said GPU is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program will be executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by one of the GPUs will wait until a storage area in the memory allocated to another program is released. A memory release method characterized by the processing being performed by a computer.

[0196] (Note 10) When a GPU shared by two or more programs is allocated to the processing of the target program, which is repeatedly executed using a machine learning model, the available memory capacity of the said GPU is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program will be executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by one of the GPUs will wait until a storage area in the memory allocated to another program is released. An information processing device characterized by having a control unit. [Explanation of Symbols]

[0197] 101 Information Processing Device 102,301 CPU 103a,103b,103,303,#1~#n,#i GPU 104,302 memory 105 Storage area 110 Target Programs 200 Information Processing Systems 201 Execution Control Unit 202 User Terminals 210 Network 220 Memory Usage Management Table 230 Free memory management table 300 bus 304 GPU memory 305 Communication I / F 306 disk drives 307 disks 308 Portable recording medium I / F 309 Portable recording media 600 Control Unit 601 Acquisition Department 602 Execution Department 603 Measuring part 604 Data Management Department 605 Allocation Section 606 Specific part 607 Judgment Department 608 Execution Control Unit 609 Memory Release Section 610 Storage section 701 Deep Learning Frameworks 702 GPU Assigner 801 Memory usage 802,803 available spaces 1401,1402 Processing image

Claims

1. When a GPU shared by two or more programs is allocated to the processing of a target program that is repeatedly executed using a machine learning model, the available memory capacity of that GPU is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program is executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by any of the GPUs will wait until a storage area in the memory allocated to another program is released. A memory release processing program characterized by having a computer perform the processing.

2. When any of the above GPUs are assigned to process the target program, it is determined whether or not that GPU is the same as the previous GPU assigned during the previous execution of the target program's processing. If it is different from the previous GPU, the memory area allocated to the target program in the memory of the previous GPU is released. The memory release processing program according to claim 1, characterized in that it causes the computer to perform the processing.

3. The aforementioned release process is, The memory release processing program according to claim 2, characterized in that it releases the memory area in the GPU memory from the previous session that was allocated to the target program and in which the model data representing the machine learning model is stored.

4. The process that performs the aforementioned execution is: The memory release processing program according to claim 2, characterized in that, if it is the same as the previous GPU, it uses model data representing the machine learning model stored in the memory of one of the GPUs to execute the processing of the target program by one of the GPUs.

5. When processing of the target program by any of the aforementioned GPUs is completed, the memory area of ​​any of the aforementioned GPUs allocated to the target program that contains data other than the model data representing the machine learning model is released. A memory release processing program according to any one of claims 1 to 4, characterized in that it causes the computer to perform the processing.

6. When a GPU shared by two or more programs is allocated to the processing of a target program that is repeatedly executed using a machine learning model, the available memory capacity of that GPU is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program is executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by any of the GPUs will wait until a storage area in the memory allocated to another program is released. A memory release method characterized by the processing being performed by a computer.

7. When a GPU shared by two or more programs is allocated to the processing of a target program that is repeatedly executed using a machine learning model, the available memory capacity of that GPU is identified. Based on a comparison of the identified free memory capacity with the memory usage measured during a previous execution of the target program, it is determined whether or not there is insufficient free memory. If there is sufficient free memory, the processing of the target program is executed by one of the GPUs. If there is insufficient free memory, the execution of the target program by any of the GPUs will wait until a storage area in the memory allocated to another program is released. An information processing device characterized by having a control unit.

Images