Control program, information processing device, and control method

Abstract

It is possible to identify bottleneck resources in the execution environment of the training program. [Solution] The information processing device includes a first control unit, a second control unit, an estimation unit, and a identification unit. The first control unit limits the performance of the first computing resource while executing a training program for a machine learning model using the first and second computing resources. After releasing the performance limit on the first computing resource, the second control unit limits the performance of the second computing resource in proportion to the limit on the first computing resource while executing the training program. The estimation unit estimates the first performance of the training program workload while the performance of the first computing resource is limited, and estimates the second performance of the training program workload while the performance of the second computing resource is limited. The identification unit identifies the bottleneck resource among the first and second computing resources based on the comparison result between the first and second performances.

Description

Technical Field

[0001] The present invention relates to a control program, an information processing apparatus, and a control method.

Background Art

[0002] For example, in a data center (DC), which is a facility that aggregates computing resources and plays an important role in supporting today's IT (Information Technology) infrastructure. DC users borrow the necessary resources from the DC and perform the required calculations. As a recent trend, there is a high demand for computing for AI (Artificial Intelligence), especially for the learning process of AI models.

[0003] Also, in the learning process of AI models, a large amount of computing resources are required, so it is often necessary to temporarily procure cloud computing resources for execution. Generally, the performance of computing resources is affected in a complex manner by various factors such as, for example, a CPU (Control Processing Unit), a GPU (Graphics Processing Unit), an accelerator, memory, storage, and a network. Therefore, the computing resources used by the user do not always match the computing demand.

[0004] [[ID=二十]]Therefore, from the perspective of DC operators, by proposing more optimal computing resources to customers who are DC users executing AI learning, the added value of the services provided by DC operators can be improved.

[0005] That is, there is a need for a mechanism to find the bottleneck resources when a customer executes AI learning, especially to find out which of the CPU and GPU becomes the bottleneck resource and propose it to the customer. At this time, it is desirable not to change the program of the AI learning executed by the customer. That is, there is a need for a method to identify the bottleneck resources in the execution environment of the AI learning application without changing the program of the AI learning executed by the customer.

[0006] Therefore, one specific method involves utilizing the fact that AI learning applications are primarily composed of loops, and estimating the application's loop period from changes in CPU usage. Specifically, the loop period of the application when the CPU operating frequency is reduced by a predetermined amount is compared with the loop period of the application when the GPU operating frequency is reduced by a predetermined amount. Based on the comparison of loop periods, the resource that causes the AI ​​learning application to perform worse, i.e., the one with the longer loop period, is identified as the bottleneck resource. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Japanese Patent Publication No. 2021-140264 [Patent Document 2] Japanese Patent Publication No. 2010-250689 [Patent Document 3] Japanese Patent Publication No. 2018-84994 [Patent Document 4] Japanese Patent Application Publication No. 7-281908 [Patent Document 5] Japanese Patent Publication No. 2010-218000 [Overview of the project] [Problems that the invention aims to solve]

[0008] One method for identifying bottleneck resources involves changing the CPU's operating frequency to pinpoint the bottleneck resource during application execution.

[0009] However, there are cases where software cannot change the CPU's operating frequency. For example, depending on the CPU's operating mode, the operating system (OS) may not be able to freely change the CPU's operating frequency.

[0010] Furthermore, changing the CPU's operating mode requires modifying the BIOS (Basic Input / Output System) settings. However, it's difficult to restart and change the BIOS settings in an already running environment. Moreover, the BIOS cannot be accessed in cloud environments, for example.

[0011] Furthermore, even if it's possible to change the CPU's operating frequency, the available frequencies are selected from a limited number of options, meaning that the optimal frequency cannot always be chosen.

[0012] Therefore, there is a need for a method to identify bottleneck resources in the execution environment of a machine learning model training program, which is an AI learning model, without changing the CPU's operating frequency.

[0013] One aspect of this project is to provide control programs that can identify bottleneck resources in the execution environment of machine learning model training programs. [Means for solving the problem]

[0014] In one embodiment of the control program, while a machine learning model training program is being executed using a first computing resource and a second computing resource, the processor is instructed to perform a process to estimate a first performance of the training program workload by limiting the performance of the first computing resource. After the limit on the performance of the first computing resource is removed, while the training program is being executed, the processor is instructed to perform a process to estimate a second performance of the training program workload by limiting the performance of the second computing resource in proportion to the percentage by which the performance of the first computing resource was limited. Based on the comparison result between the first and second performances, the processor is instructed to perform a process to identify the bottleneck resource among the first and second computing resources. [Effects of the Invention]

[0015] From one perspective, it is possible to identify bottleneck resources in the execution environment of a machine learning model training program.

Brief Description of the Drawings

[0016] [Figure 1] FIG. 1 is a schematic diagram showing an example of an information processing apparatus according to this embodiment. [Figure 2] FIG. 2 is an explanatory diagram showing an example of the configuration of an analysis program of the information processing apparatus. [Figure 3] FIG. 3 is a block diagram showing an example of the functional configuration of an analysis unit of the information processing apparatus. [Figure 4] FIG. 4 is an explanatory diagram showing an example of managing a process using cgroup_root and cgroup_limited. [Figure 5A] FIG. 5A is an explanatory diagram showing an example of the CPU time of a single thread. [Figure 5B] FIG. 5B is an explanatory diagram showing an example of the CPU time of a multi-thread. [Figure 6] FIG. 6 is an explanatory diagram showing an example of the CPU usage rate during the execution of a training program. [Figure 7] FIG. 7 is an explanatory diagram showing an example of the CPU load during the execution of a training program. [Figure 8] FIG. 8 is an explanatory diagram showing an example of the periodic change before and after the limitation of CPU performance. [Figure 9] FIG. 9 is an explanatory diagram showing an example of presentation information. [Figure 10] FIG. 10 is a flowchart showing an example of the processing operation of an analysis unit related to analysis processing. [Figure 11] FIG. 11 is a flowchart showing an example of the processing operation of an analysis unit related to the first estimation processing. [Figure 12] FIG. 12 is a flowchart showing an example of the processing operation of an analysis unit related to loop period estimation processing. [Figure 13] FIG. 13 is a flowchart showing an example of the processing operation of an analysis unit related to the second estimation processing. [Figure 14] FIG. 14 is an explanatory diagram showing an example of a method for obtaining the cumulative CPU time at the measurement start time and the cumulative CPU time at the measurement end time. [Figure 15] Figure 15 is a flowchart showing an example of the processing operation of the analysis unit involved in the thread count estimation process. [Figure 16] Figure 16 is an explanatory diagram showing an example of an API related to thread count estimation. [Figure 17] Figure 17 is an explanatory diagram showing an example of a cpu.stat file. [Figure 18] Figure 18 is a flowchart showing an example of the processing operation of the analysis unit involved in the cumulative CPU time acquisition process. [Figure 19] Figure 19 is an explanatory diagram showing an example of an API related to the process of obtaining cumulative CPU time. [Figure 20] Figure 20 is a flowchart showing an example of the processing operation of the analysis unit involved in a specific process. [Modes for carrying out the invention]

[0017] The embodiments of the control program and other components will be described below with reference to the drawings. However, the embodiments shown below are merely illustrative, and there is no intention to exclude various modifications or applications of technologies not explicitly shown in the embodiments. In other words, these embodiments can be implemented with various modifications without departing from their spirit. Furthermore, each figure is not intended to represent only the components shown in the figure, but may include other functions, etc.

[0018] (A) Configuration Figure 1 is a schematic diagram showing an example of the configuration of the information processing device 1 in this embodiment. The information processing device 1 shown in Figure 1 has a hardware platform 10 and a software platform 3. The information processing device 1 is, for example, a computer. The software platform 3 runs on the hardware platform 10. The software platform 3 includes, for example, an OS (Operating System), libraries, and frameworks.

[0019] On the software platform 3, the training program 5 for the AI ​​learning model and the analysis program 4A for analyzing bottleneck resources are executed. The software platform 3 provides, for example, the necessary software environment for executing the AI ​​learning model, the training program 5, and the analysis program 4A.

[0020] As shown in Figure 1, the hardware platform 10 has, as an example, a CPU 11, a GPU 12, memory 13, storage 14, and a communication interface 15 as its hardware configuration. These CPU 11, GPU 12, memory 13, storage 14, and communication interface 15 can also be referred to as hardware elements.

[0021] The CPU 11 is an example of an arithmetic processing unit that performs various control and calculations, and is also an example of a second computing resource. The CPU 11 may be connected to each block in the hardware platform 10 so as to be able to communicate with each other via a bus (not shown). The CPU 11 may also be a multiprocessor including multiple processors, a multicore processor having multiple processor cores, or a configuration having multiple multicore processors.

[0022] The GPU12 is a processing unit suitable for screen display control to output devices such as monitors, and is an example of a first computing resource. The GPU12 is also an example of a processing unit that performs various control and calculations. The GPU12 may be connected to each block within the hardware platform 10 via a bus (not shown) that enables communication between them. Among the hardware elements, the processing unit may also be called a processor element. The CPU11 and GPU12 are processor elements.

[0023] Memory 13 is an example of hardware that stores various data and program information. Examples of memory 13 include volatile memory such as DRAM (Dynamic Random Access Memory) and non-volatile memory such as PM (Persistent Memory).

[0024] Storage 14 is an example of hardware that stores various data and program information. Storage 14 stores programs executed by the CPU 11 and GPU 12, as well as data used in the execution of those programs. The programs include not only the AI ​​learning model, which is a machine learning model, but also the training program 5 and analysis program 4A of the AI ​​learning model.

[0025] Storage 14 includes various storage devices such as magnetic disk drives (HDDs) and other hard disk drives (HDDs), semiconductor drives (SSDs) and other solid state drives (SSDs), and non-volatile memory. Examples of non-volatile memory include flash memory, storage class memory (SCM), and read-only memory (ROM).

[0026] The communication interface 15 is an interface that controls the connection and communication between the hardware platform 10 and other information processing devices. For example, the communication interface 15 may include an adapter compliant with LAN (Local Area Network) such as Ethernet®, or optical communication such as FC (Fibre Channel). The adapter may also be an adapter that supports wireless, wired, or both communication methods.

[0027] For example, the hardware platform 10 may be connected to terminal devices and databases (not shown) via the communication interface 15 and the network, enabling them to communicate with each other. The program described above may be downloaded from the network via the communication interface 15 and stored in the storage 14.

[0028] The configuration of the hardware platform 10 can be changed as appropriate. For example, in the example shown in Figure 1, a case is illustrated in which there is one CPU 11, one GPU 12, one memory 13, one storage 14, and one communication interface 15, but there may be two or more of each. Furthermore, these hardware elements may be replaced with hardware elements of higher performance, and can be changed as appropriate.

[0029] At least some of the hardware elements constituting the hardware platform 10 may be provided by the cloud provider. Alternatively, the hardware platform 10 may be configured to achieve the desired performance by combining it with hardware elements provided by the cloud provider. Furthermore, a hardware platform 10 with sufficient performance to run the training program 5 may be configured in combination with hardware elements provided by the cloud provider.

[0030] Training program 5 is a program that trains the AI ​​learning model. Various known training programs may be used as training program 5, and their explanation is omitted. Training program 5 is executed using hardware platform 10.

[0031] Training program 5 uses multiple computing resources, specifically CPU 11 and GPU 12, to train a machine learning model.

[0032] The analysis program 4A implements a bottleneck analysis function that identifies whether the CPU 11 or the GPU 12 is the bottleneck resource during the execution of the training program 5 using the hardware platform 10. The CPU 11 and GPU 12 are computing resources that affect the execution performance of the training program 5.

[0033] Figure 2 is an explanatory diagram showing an example of the configuration of the analysis program 4A of the information processing device 1. The analysis program 4A shown in Figure 2 includes a first control program 4A1, a second control program 4A2, a measurement program 4A3, an estimation program 4A4, a specific program 4A5, and a presentation program 4A6.

[0034] The first control program 4A1 is a program that limits the performance of the GPU 12. The second control program 4A2 is a program that limits the performance of the CPU 11. The measurement program 4A3 is a program that measures the usage rate of the CPU 11. The estimation program 4A4 is a program that estimates the first loop period and the second loop period, which will be described later. The identification program 4A5 is a program that identifies the bottleneck resource based on the comparison result of the estimated first loop period and the second loop period. The presentation program 4A6 is a program that presents the results of the bottleneck resource identification to the user.

[0035] Figure 3 is a block diagram showing an example of the functional configuration of the analysis unit 4 of the information processing device 1. The analysis unit 4 shown in Figure 3 has the functions of a first control unit 41, a second control unit 42, a measurement unit 43, an estimation unit 44, a specific unit 45, and a presentation unit 46.

[0036] The CPU 11 functions as an analysis unit 4 by executing the analysis program 4A. The CPU 11 also functions as a first control unit 41 by executing the first control program 4A1. The CPU 11 functions as a second control unit 42 by executing the second control program 4A2. The CPU 11 functions as a measurement unit 43 by executing the measurement program 4A3. The CPU 11 functions as an estimation unit 44 by executing the estimation program 4A4. The CPU 11 functions as a identification unit 45 by executing the identification program 4A5. The CPU 11 functions as a presentation unit 46 by executing the presentation program 4A6.

[0037] Furthermore, the analysis program 4A does not necessarily need to include all of the first control program 4A1, the second control program 4A2, the measurement program 4A3, the estimation program 4A4, the identification program 4A5, and the presentation program 4A6. For example, at least a portion of the first control program 4A1, the second control program 4A2, the measurement program 4A3, the estimation program 4A4, the identification program 4A5, and the presentation program 4A6 may be provided outside of the analysis program 4A, and the analysis program 4A may call and execute such externally provided programs.

[0038] Furthermore, the analysis program 4A for realizing the functions of the first control unit 41, the second control unit 42, the measurement unit 43, the estimation unit 44, the identification unit 45, and the presentation unit 46 is provided, for example, in the form recorded on a computer-readable recording medium. The storage medium is, for example, a flexible disk, CD (CD-ROM, CD-R, CD-RW, etc.), DVD (DVD-ROM, DVD-RAM, DVD-R, DVD+R, DVD-RW, DVD+RW, HD DVD, etc.), Blu-ray disc, magnetic disk, optical disk, magneto-optical disk, or other recording medium. The CPU 11 reads the program from the recording medium and transfers and stores the read program in an internal storage device such as memory 13 or an external storage device such as storage 14 for use. Alternatively, the program may be recorded on a recording medium such as a magnetic disk, optical disk, or magneto-optical disk and provided to the computer via a communication path from that recording medium, and this can be changed as appropriate.

[0039] When implementing the functions of the first control unit 41, the second control unit 42, the measurement unit 43, the estimation unit 44, the identification unit 45, and the presentation unit 46, the program stored in the memory 13 is executed by the computer's CPU 11. In this case, the computer may also read and execute the program recorded on the recording medium.

[0040] [First control unit 41] The first control unit 41 controls the performance of the GPU 12. Specifically, the first control unit 41 implements a function to change (set) the operating frequency of the GPU 12. For example, the first control unit 41 changes the operating frequency of the GPU 12 by executing a command to set the operating frequency of the GPU 12.

[0041] The first control unit 41 may, for example, set the operating frequency of the GPU 12 manufactured by NVIDIA (registered trademark) using the nvidia-smi command. For example, the first control unit 41 can set the operating frequency of the NVIDIA GPU 12 to 866MHz by executing the following command.

[0042] # nvidia-smi -ac 1593,866

[0043] The first control unit 41 limits the performance of the GPU 12 by, for example, reducing the operating frequency of the GPU 12 by a predetermined amount, for example, 20%, i.e., clocking it down, for a predetermined period, for example, 20 seconds.

[0044] Furthermore, the first control unit 41 can use a similar command to return the reduced operating frequency of the GPU 12 to its original state, that is, the operating frequency before the performance limit was imposed.

[0045] [Second control unit 42] The second control unit 42 controls the performance of the CPU 11. The second control unit 42 limits the performance of the CPU 11, for example, by using the cgroup (control groups) function in Linux®. cgroup is a Linux® kernel function that restricts and isolates the use of resources by process groups, such as the CPU 11 and memory 13. In other words, cgroup restricts the allocation of resources such as the CPU 11 and memory 13 to specific processes or threads.

[0046] Figure 4 is an explanatory diagram illustrating an example of managing processes using cgroup_root and cgroup_limited. The cgroup function includes cgroup_limited, which manages processes whose performance is limited by the CPU 11, and cgroup_root, which manages processes whose performance is not limited by the CPU 11. The second control unit 42 limits the performance of the CPU 11 using cgroup_limited.

[0047] Figure 5A is an explanatory diagram illustrating an example of single-threaded CPU time. CPU performance is specified by the amount of CPU time allocated within a fixed time interval, for example, 100ms (milliseconds). For example, if 80ms of CPU time is allocated within a 100ms allocation interval, the CPU performance will be limited to 80%, which is essentially the same as limiting the operating frequency of CPU 11 to 80% (a 20% reduction). Furthermore, the CPU resource limitation applies to the entire cgroup.

[0048] Figure 5B is an explanatory diagram showing an example of multithreaded CPU time. If CPU 11 is using multiple threads, for example 2 threads, and allocates 80ms × 2 = 160ms of CPU time within a 100ms allocation interval, then the CPU performance will be limited to 80%.

[0049] The second control unit 42 limits the CPU performance so that it results in a performance reduction equal to the same limit as the GPU performance (for example, 20%).

[0050] The second control unit 42 needs to assign the training program 5 process to the performance-limiting cgroup_limited in order to limit CPU performance using the cgroup function. Training program 5 is often run in a closed environment called Docker. Note that a dedicated cgroup is allocated when Docker is started, so there is no need to create a new cgroup. The generated cgroup will be configured in the following file system directory.

[0051] / sys / fs / cgroup / system.slice / docker-<container ID>.scope

[0052] The second control unit 42 limits the CPU time allocated to the CPU 11's allocation interval according to a limiting rate, which is a percentage that limits the GPU performance, for example, a percentage that limits the operating frequency. For example, if the GPU 12, which normally operates at 1614MHz, is limited to 866MHz, the limiting rate of the GPU performance becomes 53.65%.

[0053] For example, if the workload of training program 5 uses 60 threads and there is no limit on CPU performance, then 100ms × 60 threads = 6000ms of CPU time will be allocated within the 100ms allocation interval. In contrast, if the CPU performance is limited to 53.65%, the second control unit 42 calculates a CPU time of <allocation interval × number of threads used × limit rate>, for example, <100ms × 60 threads × 53.65%> = 3219ms. The second control unit 42 then allocates the calculated 3219ms of CPU time within the allocation interval.

[0054] [Measurement unit 43] The measurement unit 43 measures the load state of the CPU 11, for example, the CPU usage rate, during the execution of the training program 5. The measurement unit 43 measures the CPU usage rate under GPU performance limitations. Specifically, the measurement unit 43 measures the CPU usage rate of the CPU 11 at 20ms intervals for 20 seconds under GPU performance limitations. As a result, the measurement unit 43 obtains the CPU usage rate for 1000 samples under GPU performance limitations.

[0055] Furthermore, the measurement unit 43 measures the CPU usage under CPU performance limitations while the training program 5 is running. Specifically, the measurement unit 43 measures the CPU usage at 20ms intervals for 20 seconds under CPU performance limitations. As a result, the measurement unit 43 obtains the CPU usage for 1000 samples under CPU performance limitations.

[0056] Figure 6 is an explanatory diagram showing an example of CPU usage during the execution of training program 5. In Figure 6, the overall system CPU usage, the CPU usage of the main thread, the CPU usage of sub-threads, and annotation information are shown for one loop included in training program 5.

[0057] The annotation information represents the loop range (loop start and end positions) within the program. In one loop included in training program 5, it can be seen that CPU usage is high because commands are being sent to GPU12. Also, near the end of one loop, CPU usage decreases as the program waits to receive the calculation results from GPU12 (inter-GPU communication). Note that CPU usage can be obtained using known methods, and their explanation is omitted here.

[0058] [Estimation part 44] The estimation unit 44 estimates the period of the loop that is repeatedly executed in the training program 5, i.e., the loop period. Based on the CPU usage under GPU performance limitations measured by the measurement unit 43, the estimation unit 44 calculates a first loop period, which is the loop period during the execution of the training program under GPU performance limitations.

[0059] Furthermore, the estimation unit 44 calculates a second loop period, which is the loop period during the execution of the training program 5 under the CPU performance limitation condition, based on the CPU usage rate under the CPU performance limitation condition equivalent to the same limitation rate as the GPU performance, as measured by the measurement unit 43.

[0060] Figure 7 is an explanatory diagram showing an example of CPU load during the execution of training program 5. In Figure 7, the horizontal axis represents the number of samples, and the vertical axis represents the CPU load (rnnt CPU load). Training programs generally consist mostly of loop execution. If the interval of one loop iteration is known, its performance (time required) can be estimated. The loop period then appears as the period of fluctuation in CPU load.

[0061] The measurement unit 43 measures the CPU usage N times at regular time intervals for a certain period, and represents the obtained measurement samples as s1, s2, ···, s N For example, assume that the measurement unit 43 measures for 20 seconds at an interval of 20 ms and obtains the CPU usage as 1000 measurement samples (in this case, N = 1000).

[0062] The estimation unit 44 obtains the autocorrelation coefficient R k while changing the lag k from 1 to N - 1 for the obtained CPU usage. Here, the autocorrelation coefficient is a statistic obtained by (Equation 1). Note that s1, s2, ···, s N are the measurement samples of the CPU usage, N is the number of measurement samples, μ is the average of the measurement samples, and σ 2 is the variance of the measurement samples.

[0063]

Equation

[0064] Among the autocorrelation coefficients obtained for each lag k, the lag that becomes the largest coefficient in the range of k > 0 is calculated. Let this be k1. The estimation unit 44 estimates the loop period (the period of CPU load fluctuation) using <k1 × (measurement interval of the load)>. Note that the measurement interval of the load is, for example, 20 ms.

[0065] FIG. 8 is an explanatory diagram showing an example of the periodic change before and after the limitation of CPU performance. In FIG. 8, the relationship between the lag and the autocorrelation calculated based on the measurement result of the CPU load during the execution of the training program 5 is shown by comparing the CPU performance before the limitation and the CPU performance after the limitation.

[0066] In FIG. 8, the relationship between the lag and the autocorrelation for the operating frequency (3.5 GHz) of the CPU performance before the limitation is shown by a solid line, and the relationship between the lag and the autocorrelation for the operating frequency (2.1 GHz) of the CPU performance after the limitation is shown by a dashed line.

[0067] This represents the estimated period (loop period) at which the peak occurs for values ​​other than k=0. In Figure 8, the waveform shifts to the right, indicating that the period lengthens. This can be seen by reducing the operating frequency of CPU11 to 2.1GHz. A longer loop period results in a longer loop execution time and a decrease in processing performance.

[0068] [Specific part 45] The identification unit 45 identifies the computational resource that is the bottleneck resource causing the performance degradation, for example, the CPU 11 or the GPU 12. The identification unit 45 compares the first loop period under the GPU performance limit and the second loop period under the CPU performance limit. Based on the comparison result between the first and second loop periods, the identification unit 45 identifies the computational resource with the longer loop period, i.e., the one that causes the word load performance to decrease, as the bottleneck resource among the CPU 11 and the GPU 12. The computational resource determined to be the bottleneck resource is the computational resource to be improved. In other words, the identification unit 45 identifies the computational resource to be improved from among the CPU 11 and the GPU 12.

[0069] [Presentation part 46] The presentation unit 46 presents the user with presentation information indicating the computing resources identified as bottleneck resources. The presentation unit 46 also presents the user with presentation information indicating the computing resources to be improved, as identified by the identification unit 45.

[0070] Figure 9 is an explanatory diagram showing an example of the information to be presented. In Figure 9, the display screen 50 is shown on the monitor of a terminal device (not shown) connected to the hardware platform 10 via a network or the like. The display screen 50 corresponds to the information to be presented output by the presentation unit 46.

[0071] Display screen 50 indicates that the analysis program 4A has been executed by running the command “analyze_training_performance” (see symbol P1).

[0072] Furthermore, the display screen 50 shows the first loop period under GPU performance limitations and the second loop period under CPU performance limitations, as estimated by the estimation unit 44 (see reference numeral P2). Specifically, the display screen 50 shows that the second loop period is 600ms and the first loop period is 540ms.

[0073] Furthermore, the display screen 50 shows a message indicating the computing resource identified as a bottleneck (see symbol P3).

[0074] On display screen 50, the second loop period (600ms) under CPU performance limitations is longer than the first loop period (540ms) under GPU performance limitations. As a result, display screen 50 shows the message "CPU performance may limit the total performance," indicating that CPU 11 is the bottleneck resource.

[0075] The information presented in Figure 9 is merely an example, and the information output by the presentation unit 46 can be modified as appropriate. For example, the presented information may include information other than that exemplified in Figure 9. Also, the second loop period under CPU performance limitations and the first loop period under GPU performance limitations, which are estimated by the estimation unit 44, may be omitted and modified as appropriate.

[0076] (B) Operation Next, the operation of the information processing device 1 in this embodiment will be described. Figure 10 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in the analysis process. For example, suppose that a DC user or a user performing the analysis is running the training program 5 to be analyzed while the analysis program 4A is running. In Figure 10, the first control unit 41 and the estimation unit 44 within the analysis unit 4 perform the first estimation process shown in Figure 11 (step S1). The first estimation process is a process that estimates the first loop period under the limitations of GPU performance.

[0077] The second control unit 42 and estimation unit 44 within the analysis unit 4 perform the second estimation process shown in Figure 13 after performing the first estimation process (step S2). The second estimation process is a process for estimating the second loop period under CPU performance limitations.

[0078] The identification unit 45 within the analysis unit 4 performs the identification process shown in Figure 20 after executing the second estimation process (step S3). The identification process identifies the bottleneck resource based on the comparison result between the first loop period and the second loop period. Then, the presentation unit 46 within the analysis unit 4, after executing the identification process, presents presentation information indicating the identified bottleneck resource to the user (step S4), and terminates the processing operation shown in Figure 10.

[0079] Furthermore, the order of processing steps S1 and S2 in the analysis process is not limited to this and can be changed as appropriate. That is, the first estimation process in step S1 may be executed after the second estimation process in step S2, or the first estimation process in step S1 and the second estimation process in step S2 may be executed in parallel.

[0080] Figure 11 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in the first estimation process. The first control unit 41 within the analysis unit 4 limits the GPU performance by reducing the operating frequency of the GPU 12 by a predetermined amount (step S11). Reducing by a predetermined amount means, for example, limiting the GPU performance by reducing it by 20% compared to the normal operating frequency of the GPU 12.

[0081] The measurement unit 43 in the analysis unit 4 measures the CPU usage rate for a predetermined time during the execution of the training program 5 under the limited state of GPU performance (step S12). The estimation unit 44 in the analysis unit 4 executes a loop period estimation process based on the measurement result of the CPU usage rate for a predetermined time (step S13). Note that in the loop period estimation process of step S13, it is a process of estimating the first loop period using the CPU usage rate for a predetermined time during the execution of the training program 5 under the limited state of GPU performance. The estimation unit 44 acquires the first loop period estimated in the loop period estimation process (step S14).

[0082] After acquiring the first loop period, the first control unit 41 returns the operating frequency of the GPU 12 to the operating frequency before the GPU performance limitation (step S15). Then, the estimation unit 44 stores the acquired first loop period in the memory 13 (step S16). Then, the first control unit 41 ends the processing operation of the first estimation process shown in FIG. 11 and shifts to the second estimation process of step S2 in FIG. 10.

[0083] FIG. 12 is a flowchart showing an example of the processing operation of the analysis unit 4 related to the loop period estimation process. The measurement unit 43 in the analysis unit 4 measures the CPU usage rate N times at a fixed time and at a fixed time interval (step S21). The estimation unit 44 in the analysis unit 4 obtains the autocorrelation coefficient R k while changing the lag k from 1 to N - 1 for the measurement samples acquired by the measurement unit 43 (step S22).

[0084] The estimation unit 44 calculates the lag (k1) that becomes the largest coefficient in the range of k > 0 among the autocorrelation coefficients obtained for each lag k. The estimation unit 44 estimates the loop period based on <k1 × measurement interval of the load> (step S24), and ends the processing operation shown in FIG. 12. Then, it shifts to the process of step S14 shown in FIG. 11 or step S38 shown in FIG. 13.

[0085] Furthermore, when the analysis unit 4 executes the loop period estimation process from step S12 of the first estimation process in Figure 11, it measures the CPU usage for a predetermined period of time under GPU performance limitations and estimates the first loop period based on the CPU usage for that predetermined period of time.

[0086] Furthermore, when the analysis unit 4 executes the loop period estimation process from step S36 of the second estimation process in Figure 13, it measures the CPU usage rate for a predetermined period of time under CPU performance limitations and estimates the second loop period based on the CPU usage rate for the predetermined period of time.

[0087] Figure 13 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in the second estimation process. The second control unit 42 within the analysis unit 4 identifies the cgroup to be assigned to the training program 5 process (step S31). The second control unit 42 calculates the limiting rate of the GPU 12 based on (operating frequency of the GPU 12 after a predetermined amount of reduction after the limiting ÷ operating frequency of the GPU 12 before the limiting) (step S32).

[0088] The second control unit 42 calculates the limiting ratio of the GPU 12 and then executes the thread count estimation process shown in Figure 14 (step S33). The thread count estimation process is, for example, the process of estimating the number of threads of the CPU 11 used for the training program 5 process.

[0089] The second control unit 42 calculates the CPU time corresponding to the performance limit of the CPU 11 based on <allocation interval × number of threads × limit rate> (step S34). The second control unit 42 limits the CPU performance by setting the calculated CPU time to the CPU 11 (step S35).

[0090] The measurement unit 43 measures the CPU usage rate for a predetermined period of time under CPU performance limitations (step S36). Then, the estimation unit 44 uses the measured CPU usage rate for the predetermined period of time to perform the loop period estimation process shown in Figure 12 (step S37). Note that the loop period estimation process in step S37 is a process that estimates the second loop period using the CPU usage rate for a predetermined period of time while the training program 5 is running under CPU performance limitations.

[0091] After performing the loop period estimation process in step S37, the estimation unit 44 obtains the second loop period (step S38).

[0092] The second control unit 42 restores the CPU time of the CPU 11 after the performance limit to the CPU time before the performance limit (step S39). The estimation unit 44 stores the acquired second loop period in the memory 13 (step S40), terminates the processing operation shown in Figure 13, and proceeds to the specific processing in step S3 of Figure 10.

[0093] Figure 14 is an explanatory diagram illustrating an example of how to obtain the cumulative CPU time at the start and end of the measurement. In Figure 4, for example, if the interval time is 100ms, the cumulative CPU time at the start of the measurement is 1000ms, and the cumulative CPU time at the end of the measurement is 16000ms, the difference in cumulative CPU time will be 6000ms.

[0094] Therefore, the second control unit 42 can calculate 60 threads as the number of threads to be used when executing the workload, based on (cumulative CPU time at the end of measurement - cumulative CPU time at the start of measurement) ÷ interval time, i.e., (6000ms ÷ 100ms).

[0095] Figure 15 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in the thread count estimation process. The second control unit 42 within the analysis unit 4 executes a cumulative CPU time acquisition process to acquire the current cumulative CPU time (step S51).

[0096] The second control unit 42 executes the cumulative CPU time acquisition process in step S51 and then acquires the cumulative CPU time at the measurement start time (step S52). After acquiring the cumulative CPU time at the measurement start time, the second control unit 42 waits for a predetermined time (step S53).

[0097] The second control unit 42 executes a cumulative CPU acquisition process to acquire the current cumulative CPU time after waiting for a predetermined time (step S54). After executing the cumulative CPU time acquisition process in step S54, the second control unit 42 acquires the cumulative CPU time at the measurement end time (step S55).

[0098] The second control unit 42 obtains the cumulative CPU time at the end of the measurement and then calculates the number of threads for the workload of the training program 5 based on (<cumulative CPU time at the end of the measurement - cumulative CPU time at the start of the measurement> ÷ interval time) (step S56). The second control unit 42 stores the calculated number of threads in the memory 13 (step S57) and terminates the processing operation shown in Figure 15.

[0099] Figure 16 is an explanatory diagram showing an example of an API related to thread count estimation. Cumulative CPU time can be obtained in various programming languages. For example, in C++, it can be obtained using the API (Application Programming Interface) shown in Figure 16.

[0100] The second control unit 42 obtains the measurement start time (P11). The second control unit 42 obtains the cumulative CPU time at the measurement start time (P12). The second control unit 42 waits for a predetermined time from the measurement start time (P13). The second control unit 42 obtains the measurement end time (P14). The second control unit 42 obtains the cumulative CPU time at the measurement end time (P15). The second control unit 42 obtains the cumulative CPU time which is the difference between (cumulative CPU time at measurement end time - cumulative CPU time at measurement start time) (P16). The second control unit 42 calculates the number of threads based on (cumulative CPU time ÷ interval time) (P17).

[0101] Figure 17 is an explanatory diagram showing an example of a cpu.stat file. The cpu.stat file is written in the format shown in Figure 17. The first part of the file path, / sys / fs / cgroup / system.slice / docker-<container ID>, is the path to the cgroup to which training program 5 is assigned. Other container systems (such as podman or slurm) may be assigned to a different path. The last part of the file path, "cpu.stat", is common regardless of the type of container system. Cumulative CPU time is updated cumulatively in the field called "usage_usec".

[0102] The cgroup function allocates CPU time to an allocation interval by deploying it on the file system and performing read / write operations on specific files. CPU time limits are limited by write operations on the following files.

[0103] The cgroup settings are configured by writing the following values ​​to the aforementioned file. The units for CPU time and allocation intervals are microseconds (μs). As mentioned earlier, the units for CPU time and allocation intervals are milliseconds (ms), so the value of CPU time to be allocated within the allocation interval is multiplied by 1000.

[0104] Furthermore, to write the CPU time to be allocated within the allocation interval using a shell script, combine the sudo and echo commands as follows:

[0105] $ sudo sh -c “echo '3219000 100000' > \ / sys / fs / cgroup / system.slice / docker-<container ID>.scope / cpu.max”

[0106] Furthermore, to remove the CPU performance limit, specify "max" for the "CPU time" allocated within the allocation interval. Then, when writing the CPU time within the allocation interval using a shell script, it would look like this:

[0107] $ sudo sh -c “echo 'max 100000' > \ / sys / fs / cgroup / system.slice / docker-<container ID>.scope / cpu.max”

[0108] The second control unit 42 obtains the cumulative CPU time at the start of measurement from the measurement start time "usage_usec" in the cpu.stat file. The second control unit 42 also obtains the cumulative CPU time at the end of measurement from the measurement end time "usage_usec" in the cpu.stat file. The second control unit 42 obtains the cumulative CPU time which is the difference between the cumulative CPU time at the end of measurement and the cumulative CPU time at the start of measurement.

[0109] Figure 18 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in the cumulative CPU time acquisition process. The second control unit 42 within the analysis unit 4 opens the cpu.stat file (step S61) and reads one line from the opened file (step S62).

[0110] The second control unit 42 splits the read line into words, making the first word a string and the second word a number (step S63). The second control unit 42 determines whether the first word is "usage_usec" (step S64). If the first word is "usage_usec" (step S64: Yes), the second control unit 42 obtains the second word as the cumulative CPU time (step S65) and terminates the processing operation shown in Figure 18.

[0111] If the first word is not "usage_usec" (step S64: No), the second control unit 42 reads the next line (step S66) and returns to step S63, which splits the words in the line.

[0112] Figure 19 is an explanatory diagram showing an example of an API related to the cumulative CPU time acquisition process. The second control unit 42 can acquire the cumulative CPU time via the file system. In C++, the program is as shown in Figure 19. Specify the path to the aforementioned cpu.stats file in the filename part.

[0113] The second control unit 42 opens the cpu.stat file (P21). The second control unit 42 reads one line from the opened file (P22). The second control unit 42 splits the read line into words, treating the first word as a string and the second word as a number (P23).

[0114] The second control unit 42, if the first word is "usage_usec" (P24), obtains the second word as the cumulative CPU time (P25) and returns the cumulative CPU time (P26).

[0115] Figure 20 is a flowchart showing an example of the processing operation of the analysis unit 4 involved in a specific process. The specific process shown in Figure 20 is the specific process of step S3 in Figure 10. The specific unit 45 within the analysis unit 4 compares the first loop period and the second loop period (step S71). Based on the comparison result, the specific unit 45 determines whether the first loop period is longer or not (step S72).

[0116] If the first loop period is longer (step S72: Yes), the specific unit 45 determines that the GPU 12 is the bottleneck computing resource (step S73), terminates the processing operation shown in Figure 20, and proceeds to the processing in step S4 of Figure 10.

[0117] If the first loop period is not longer (step S72: No), the specific unit 45 determines that the second loop period is longer and that the CPU 11 is the bottleneck computing resource (step S74), and terminates the processing operation shown in Figure 20. Then, it proceeds to the processing in step S4 of Figure 10.

[0118] (C) Effects In this embodiment, the estimation unit 44 of the information processing device 1 estimates the first loop period during the execution of the training program 5 under GPU performance limitations. Furthermore, the second control unit 42 uses the cgroup function to limit CPU performance according to the number of threads used and the GPU performance limitation rate without changing the operating frequency. Then, after releasing the GPU performance limitation, the estimation unit 44 estimates the second loop period during the execution of the training program 5 under CPU performance limitations. The identification unit 45 compares the first loop period and the second loop period and identifies the computing resource with the longer loop period among the CPU 11 and GPU 12 as the bottleneck resource. As a result, the bottleneck resource under the execution environment of the training program 5 can be identified without changing the operating frequency of the CPU 11. Regarding the execution of the training program 5, the computing resource that becomes a bottleneck can be easily identified, and an optimal computing configuration for executing the training program 5 can be constructed. As a result, the training efficiency of the machine learning model can be improved.

[0119] The first control unit 41 limits the GPU performance by reducing the operating frequency of the GPU 12 by a predetermined amount. The second control unit 42 calculates the limiting rate of the GPU 12 when the operating frequency of the GPU 12 is reduced by a predetermined amount, and limits the performance of the CPU 11 according to the limiting rate. The estimation unit 44 estimates the first loop period while the training program 5 is running under the GPU performance limiting condition, and estimates the second loop period while the training program 5 is running under the CPU performance limiting condition. As a result, bottleneck resources in the execution environment of the training program 5 can be identified without changing the operating frequency of the CPU 11.

[0120] The second control unit 42 limits CPU performance by shortening the CPU time allocated to the training program for each allocation interval according to the limit rate. As a result, CPU performance can be limited without changing the operating frequency of the CPU 11.

[0121] The second control unit 42 limits CPU performance by shortening CPU time based on the GPU 12 limit rate, allocation interval, and the number of threads of the CPU 11 used for the training program 5. As a result, CPU performance can be limited without changing the operating frequency of the CPU 11.

[0122] The second control unit 42 limits CPU performance by shortening the CPU time according to the limit rate of the GPU 12 using the cgroup function. As a result, CPU performance can be limited without changing the operating frequency of the CPU 11.

[0123] The presentation unit 46 presents information including the identified bottleneck resources. As a result, the user can easily identify the bottleneck resources in the execution environment of the training program 5 without changing the operating frequency of the CPU 11, and grasp the optimal computer configuration for running the training program 5.

[0124] The method of limiting CPU performance using the cgroup function allows for performance restriction even in environments where the operating frequency of CPU11 cannot be set in software. Furthermore, the cgroup function allows for more granular performance restriction compared to changing the operating frequency of CPU11. Additionally, the cgroup function allows for CPU performance restriction on a per-application basis.

[0125] Furthermore, limiting CPU performance by changing the operating frequency affects the overall system performance. This impacts processes other than applications, such as network and storage data processing. As a result, it becomes difficult to distinguish whether the slowdown is due to the application or other factors, making it challenging to pinpoint the bottleneck. In contrast, limiting CPU performance using the cgroup function limits only the CPU performance within the cgroup to which the application is assigned. Consequently, if the CPU performance used by the application decreases, it can be clearly concluded that the CPU portion of the application's processing is the bottleneck.

[0126] Furthermore, from the perspective of a cloud provider, it is highly convenient because they can easily identify bottleneck computing resources using CPU usage data, which is observable by the cloud provider, without having to modify the customer's program. In addition, by presenting the bottleneck computing resources to the customer, it is possible to improve the added value of the service and increase customer satisfaction.

[0127] The estimation unit 44 can easily estimate the loop period by using the autocorrelation coefficient.

[0128] (D) Other The disclosed technology is not limited to the embodiments described above and can be implemented in various modifications without departing from the spirit of this embodiment. Each configuration and process of this embodiment can be selected or combined as needed.

[0129] For example, in the embodiment described above, in a hardware platform 10 equipped with two processor elements (computational resources), a CPU 11 and a GPU 12, it is determined which of the CPU 11 or the GPU 12 is the bottleneck (the resource to be improved). However, the bottleneck resource is not limited to the CPU 11 and the GPU 12 and can be changed as appropriate.

[0130] The system may also include computing resources other than CPUs and GPUs, such as MPUs (Micro Processing Units) and APUs (Accelerated Processing Units). Furthermore, it may have three or more computing resources, and identify the bottleneck (the resource to be improved) from among these three or more resources.

[0131] Furthermore, the same method can be applied to hardware elements other than the computation elements to identify the hardware elements that are causing bottlenecks.

[0132] Furthermore, in the embodiment described above, the CPU 11 that executes the training program 5 also executes the analysis program 4A, but this is not limited to this. The analysis program 4A may also be executed by a processor prepared separately from the CPU 11, or by a processor installed in a computer provided separately from the hardware platform 10.

[0133] Furthermore, in the above-described embodiment, the estimation unit 44 estimates the loop period of the training program 5 using the autocorrelation coefficient, but is not limited to this. For example, the loop period may be estimated using other known methods, such as using a Fourier series. Moreover, this embodiment can be implemented and manufactured by those skilled in the art based on the above disclosure.

[0134] (E) Note The following additional information is disclosed regarding the embodiments described above.

[0135] [program] (Note 1) While the machine learning model training program is running using the first and second computing resources, the performance of the first computing resource is limited to estimate the first performance of the training program's workload. After removing the performance limit on the first computing resource, the performance of the second computing resource is limited in proportion to the rate at which the performance of the first computing resource was limited during the execution of the training program to estimate the second performance of the training program's workload. Based on the comparison results between the first performance and the second performance, the bottleneck resource among the first and second computing resources is identified. A control program characterized by causing a processor to execute a process.

[0136] (Note 2) The process for identifying the bottleneck resource is as follows: The control program according to Appendix 1, characterized in that it identifies the first or second computing resource, which is limited in order to obtain the lower of the first and second performances, as a bottleneck resource.

[0137] (Note 3) The process for estimating the first performance is as follows: The performance of the first computing resource is estimated by limiting its performance by reducing its operating frequency by a predetermined amount. The process for estimating the second performance described above is: The control program according to Appendix 1, characterized in that it calculates the limiting rate of the first computing resource when the operating frequency of the first computing resource is reduced by a predetermined amount, and estimates the performance of the second computing resource by limiting its performance according to the limiting rate using the cgroup function.

[0138] (Note 4) The process that limits the performance of the second computing resource is: The control program according to Appendix 3, characterized in that it limits the performance of the second computing resource by shortening the operating time of the second computing resource allocated to the training program for each predetermined allocation interval according to the limit rate.

[0139] (Note 5) The process of limiting the performance of the second computing resource is: The control program according to Appendix 4, characterized in that it limits the performance of the second computing resource by shortening the operating time of the second computing resource based on the limiting rate of the first computing resource, the allocation interval, and the number of threads of the second computing resource used for the training program.

[0140] (Note 6) The process for estimating the first performance is as follows: The first loop period for executing the workload of the training program while limiting the performance of the first computing resource is estimated as the first performance. The process for estimating the second performance described above is: The control program according to Appendix 1, characterized in that it uses the cgroup function to estimate a second loop period as the second performance for executing the workload of the training program while limiting the performance of the second computing resource.

[0141] (Note 7) The control program according to Note 1, characterized in that the first computing resource is a GPU (Graphics Processing Unit) and the second computing resource is a CPU (Control Processing Unit).

[0142] (Note 8) The identified bottleneck resource is presented. The control program according to Appendix 1, characterized in that it causes the processor to perform further processing.

[0143] (Note 9) A first control unit that limits the performance of the first computing resource while a machine learning model training program is being executed using the first computing resource and the second computing resource, After removing the performance limit on the first computing resource, a second control unit limits the performance of the second computing resource during the execution of the training program in proportion to the percentage to which the performance of the first computing resource was limited, An estimation unit which estimates the first performance of the training program workload while limiting the performance of the first computing resource in the first control unit, and estimates the second performance of the training program workload while limiting the performance of the second computing resource in the second control unit, Based on the comparison results of the first performance and the second performance, an identification unit identifies the bottleneck resource among the first and second computing resources, An information processing device characterized by having the following features.

[0144] (Note 10) The specified part is, The information processing apparatus according to Appendix 9, characterized in that it identifies the first or second computing resource, which is limited in order to obtain the lower of the first and second performances, as a bottleneck resource.

[0145] (Note 11) The first control unit is, The performance of the first computing resource is limited by reducing the operating frequency of the first computing resource by a predetermined amount. The second control unit, The information processing device according to Appendix 10, characterized in that it calculates the limiting rate of the first computing resource when the operating frequency of the first computing resource is reduced by a predetermined amount, and uses the cgroup function to limit the performance of the second computing resource according to the limiting rate.

[0146] (Note 12) An information processing device that executes a training program for a machine learning model using the first computing resource and the second computing resource, During the execution of the training program, the performance of the first computing resource is limited to estimate the first performance of the training program's workload. After removing the performance limit on the first computing resource, the performance of the second computing resource is limited in proportion to the rate at which the performance of the first computing resource was limited during the execution of the training program to estimate the second performance of the training program's workload. Based on the comparison results between the first performance and the second performance, the bottleneck resource among the first and second computing resources is identified. A control method characterized by executing a process.

[0147] (Note 13) The process of identifying the bottleneck resource is as follows: The control method according to Appendix 12, characterized in that the first computing resource or the second computing resource, which is limited in order to obtain the lower of the first and second performances, is identified as a bottleneck resource.

[0148] (Note 14) The process for estimating the first performance is as follows: The performance of the first computing resource is estimated by limiting its performance by reducing its operating frequency by a predetermined amount. The process for estimating the second performance described above is: The control method according to Appendix 12, characterized in that the limiting rate of the first computing resource is calculated when the operating frequency of the first computing resource is reduced by a predetermined amount, and the performance of the second computing resource is estimated by limiting the performance of the second computing resource according to the limiting rate using the cgroup function. [Explanation of symbols]

[0149] 1. Information Processing Device 4 Analysis section 4A Analysis Program 5 Training Program 11 CPU 12 GPU 41 First control unit 42 Second control unit 43 Measurement Unit 44 Estimation part 45 Specific part 46 Presentation section 4A1 First control program 4A2 Second control program 4A3 Measurement Program 4A4 Estimation Program 4A5 Specific Program 4A6 Presentation Program

Claims

1. During the execution of a machine learning model training program using the first and second computing resources, the performance of the first computing resource is limited to estimate the first performance of the training program's workload. After removing the performance limit on the first computing resource, the performance of the second computing resource is limited in proportion to the percentage to which the performance of the first computing resource was limited during the execution of the training program to estimate the second performance of the training program's workload. Based on the comparison results between the first performance and the second performance, the bottleneck resource among the first and second computing resources is identified. A control program characterized by causing a processor to execute a process.

2. The process of identifying the aforementioned bottleneck resource is: The control program according to claim 1, characterized in that it identifies the first or second computing resource, which is limited in order to obtain the lower of the first and second performances, as a bottleneck resource.

3. The process for estimating the first performance is as follows: The performance of the first computing resource is estimated by limiting its performance by reducing its operating frequency by a predetermined amount. The process for estimating the second performance described above is: The control program according to claim 1, characterized in that it calculates the limiting rate of the first computing resource when the operating frequency of the first computing resource is reduced by a predetermined amount, and estimates the performance of the second computing resource by limiting its performance according to the limiting rate using the cgroup function.

4. The process of limiting the performance of the second computing resource is: The control program according to claim 3, characterized in that it limits the performance of the second computing resource by shortening the operating time of the second computing resource allocated to the training program for each predetermined allocation interval according to the limit rate.

5. The process of limiting the performance of the second computing resource is: The control program according to claim 4, characterized in that it limits the performance of the second computing resource by shortening the operating time of the second computing resource based on the limiting rate of the first computing resource, the allocation interval, and the number of threads of the second computing resource used for the training program.

6. The process for estimating the first performance is as follows: The first loop period for executing the workload of the training program while limiting the performance of the first computing resource is estimated as the first performance. The process for estimating the second performance described above is: The control program according to claim 1, characterized in that it uses the cgroup function to estimate the second loop period for executing the workload of the training program while limiting the performance of the second computing resource as the second performance.

7. The control program according to claim 1, characterized in that the first computing resource is a GPU (Graphics Processing Unit) and the second computing resource is a CPU (Control Processing Unit).

8. The identified bottleneck resource is presented. The control program according to claim 1, characterized in that it causes the processor to perform further processing.

9. A first control unit limits the performance of the first computing resource while a machine learning model training program is being executed using the first and second computing resources, After removing the performance limit on the first computing resource, a second control unit limits the performance of the second computing resource in proportion to the percentage to which the performance of the first computing resource was limited during the execution of the training program, An estimation unit which estimates the first performance of the training program workload while limiting the performance of the first computing resource in the first control unit, and estimates the second performance of the training program workload while limiting the performance of the second computing resource in the second control unit, Based on the comparison results of the first performance and the second performance, an identification unit identifies the bottleneck resource among the first and second computing resources, An information processing device characterized by having the following features.

10. An information processing device that executes a training program for a machine learning model using a first computing resource and a second computing resource, During the execution of the training program, the performance of the first computing resource is limited to estimate the first performance of the training program's workload. After removing the performance limit on the first computing resource, the performance of the second computing resource is limited in proportion to the percentage to which the performance of the first computing resource was limited during the execution of the training program to estimate the second performance of the training program's workload. Based on the comparison results between the first performance and the second performance, the bottleneck resource among the first and second computing resources is identified. A control method characterized by executing a process.

Images