System, management base, program, and scheduling method

Abstract

Provided is a system comprising: a plurality of distribution bases that execute distributed learning for one learning task; and a management base that manages the plurality of distribution bases. Each of the plurality of distribution bases includes: a metrics transmission unit that transmits metrics of each of the distribution bases to a scheduler; and an execution unit that executes a part of the one learning task. The management base includes: a distributed learning determination unit that determines, on the basis of metrics of the plurality of distribution bases, a DP group using DP, a PP group using PP, and a TP group using TP among the plurality of distribution bases; and a scheduling unit that perform scheduling so as to cause the DP group to execute a part of the one learning task by using DP, cause the PP group to execute a part of the one learning task by using PP, and cause the TP group to execute a part of the one learning task by using TP. The distribution base may execute RAN control and AI processing (including RAN control AI processing, such as a RAN intelligent controller (RIC), and non-RAN control AI processing).

Description

System, management infrastructure, program, and scheduling method

[0001] The present invention relates to a system, a management base, a program, and a scheduling method.

[0002] Patent Document 1 describes an information processing device including a management unit that manages workers available for distributed learning of a model, and a parameter server unit that distributes tasks to the workers managed by the management unit, collects gradient data obtained by learning by the workers, and updates the model. [Prior art documents] [Patent documents] [Patent document 1] International Publication No. 2021 / 220616

[0003] Common supervised distributed learning methods include data parallel (sometimes referred to as DP) and model parallel (sometimes referred to as MP). Model parallel includes pipeline parallel (sometimes referred to as PP) and tensor parallel (sometimes referred to as TP). For example, when performing distributed learning using multiple GPUs (Graphics Processing Units), data parallel divides a data set into the number of GPUs and assigns it to each GPU, with each GPU having its own copy of the model and learning the assigned data set separately. Then, for example, after backpropagation, synchronization is achieved by sharing the gradient of the model between the GPUs. In pipeline parallelism, the model is divided into layer bundles according to the number of GPUs, and layer bundles are assigned to each GPU for training. In tensor parallelism, tensors are divided into the number of GPUs and assigned to each GPU, and each GPU processes the assigned tensor.

[0004] DP, PP, and TP have a trade-off between communication volume and memory efficiency. Generally, TP requires more communication volume in the order of TP, PP, and DP. That is, among DP, PP, and TP, TP requires the most communication volume. Also, generally, DP, PP, and TP require the most total memory volume in the order of DP, PP, and TP. That is, among TP, PP, and DP, DP requires the most total memory volume across the entire distributed learning execution platform. 3D parallelism learning divides learning tasks into three dimensions while making good use of these. In order to determine this 3D parallelism strategy, it has traditionally been assumed that the GPU resources available during learning are known, and they are assigned or searched for in advance.

[0005] In the system according to this embodiment, the DP, PP, and TP are dynamically changed according to the situation, thereby contributing to the efficient use of learning resources. As a specific example, the system according to this embodiment includes a distributed infrastructure having a RAN control function for controlling a RAN (Radio Area Network) and an AI processing function for performing AI (Artificial Intelligence) processing, and a management infrastructure for managing multiple distributed infrastructures, and determines, from the multiple distributed infrastructures, a DP group that uses DP, a PP group that uses PP, and a TP group that uses TP based on metrics of the multiple distributed infrastructures.

[0006] Types of AI processing include AI processing related to RAN control (sometimes referred to as RAN-controlled AI processing) and AI processing not related to RAN control (sometimes referred to as non-RAN-controlled AI processing).

[0007] An example of RAN control AI processing is RIC (RAN Intelligent Controller). RIC is a technology that uses AI to optimize RAN radio resources and automate RAN operations. RIC includes Non-RT RIC (Non-Real Time RIC) and Near-RT RIC (Near-Real Time RIC). Non-RT RIC is sometimes called Centralized RIC. Non-RT RIC is located inside SMO (Service Management and Orchestration), which manages and orchestrates the RAN. Non-RT RIC generates and notifies policies related to RAN control and sends information to Near-RT RIC. For example, the Non-RT RIC performs machine learning using data collected from the RAN to generate a trained model for RAN control and transmits it to the Near-RT RIC. The Near-RT RIC is sometimes called a Distributed RIC. Compared to the Non-RT RIC, the Near-RT RIC is located closer to the RAN nodes (RU (Radio Unit), DU (Distributed Unit), CU (Central Unit)) and controls the RAN nodes, resources, etc. The Near-RT RIC performs processing with higher real-time performance than the Non-RT RIC. The Near-RT RIC performs inference processing related to RAN control using, for example, a trained model acquired from the Non-RT RIC. RAN control AI processing is not limited to the RIC.

[0008] The non-RAN control AI processing may correspond to a so-called MEC (Multi-access Edge Computing) application. Examples of the non-RAN control AI processing include a monitoring AI execution process that determines the situation within the imaging range of an input captured image, and a response AI execution process that outputs a response to an input user inquiry. However, this is not limited to these.

[0009] In 3D parallel learning, optimizing the division strategy enables optimal learning, primarily for speed. However, conventional methods assume that the instances used during learning are known. Once learning based on that strategy begins, the learning instances cannot be released. This makes them unsuitable for use in distributed infrastructures where supply and demand change dynamically, such as systems that control RANs consisting of a management platform and multiple distributed platforms. The system according to this embodiment introduces a scheduler that can divide learning tasks into jobs at the epoch (or layer) level and schedule them to computing resources. By dynamically allocating jobs to distributed platforms during learning, low-cost learning can be achieved by utilizing available resources on the distributed platforms.

[0010] According to one embodiment of the present invention, a system is provided. The system may include a plurality of distributed platforms that perform distributed learning for a single learning task. The system may include a management platform that manages the plurality of distributed platforms. Each of the plurality of distributed platforms may include a metrics sending unit that sends metrics of the distributed platform to the scheduler. Each of the plurality of distributed platforms may include an execution unit that executes a portion of the single learning task. The management platform may include a distributed learning determination unit that determines, based on the metrics of the plurality of distributed platforms, a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism. The management platform may include a scheduling unit that schedules the DP group to execute a portion of the single learning task using data parallelism, the PP group to execute a portion of the single learning task using pipeline parallelism, and the TP group to execute a portion of the single learning task using tensor parallelism.

[0011] In the system, the execution unit may include a plurality of execution entities, the metrics sending unit may send metrics of each of the plurality of execution entities to the management infrastructure, and the classifying unit may classify the plurality of execution entities of the plurality of distributed infrastructures into the DP group, the PP group, and the TP group based on the metrics of each of the plurality of execution entities of the plurality of distributed infrastructures. The plurality of execution entities may include a plurality of GPU instances and a plurality of GPUs, and the metrics sending unit may send metrics of the plurality of GPUs and metrics of the plurality of GPU instances to the management infrastructure.

[0012] In any of the above systems, the metrics may indicate communication conditions between the executing entities, and the classifier may include, among the multiple executing entities, multiple executing entities whose degree of goodness of their communication conditions with each other is greater than a predetermined first degree in the TP group, multiple executing entities whose degree of goodness of their communication conditions with each other is less than the first degree and greater than a second degree that is less than the first degree in the PP group, and multiple executing entities whose degree of goodness of their communication conditions with each other is less than the second degree in the DP group. The metrics may further indicate usage conditions of computational resources of the executing entities, and the classifier may classify the multiple executing entities into the DP group, the PP group, and the TP group based on the communication conditions between the executing entities and the usage conditions of the computational resources of the executing entities.

[0013] In any of the above systems, the metrics sending unit may send metrics of the execution units of the distributed infrastructure to the management infrastructure for each epoch of the one learning task, the classification unit may classify the execution units of the multiple distributed infrastructures into the DP group, the PP group, and the TP group based on the metrics of the multiple distributed infrastructures for each epoch of the one learning task, and the scheduling unit may schedule the DP group to execute a portion of the one learning task in data parallel, the PP group to execute a portion of the one learning task in pipeline parallel, and the TP group to execute a portion of the one learning task in tensor parallel, for each epoch of the one learning task.

[0014] In any of the systems described above, the execution unit may execute a RAN control function that controls a RAN and a part of the one learning task using computational resources of the distributed infrastructure. If there is a part of the computational resources of the distributed infrastructure that is not being used by the RAN control function, the execution unit may execute the part of the one learning task using the part.

[0015] According to one embodiment of the present invention, a management platform is provided. The management platform may include a metrics acquisition unit that acquires metrics of a plurality of distributed platforms from each of the plurality of distributed platforms that execute distributed learning for a single learning task. The management platform may include a classification unit that classifies the execution units of the plurality of distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism based on the metrics of the plurality of distributed platforms. The management platform may include a scheduling unit that schedules the DP group to execute a portion of the single learning task using data parallelism, the PP group to execute a portion of the single learning task using pipeline parallelism, and the TP group to execute a portion of the single learning task using tensor parallelism.

[0016] According to one embodiment of the present invention, a program is provided for causing a computer to execute the following steps: an acquisition step for acquiring metrics of a plurality of distributed platforms from each of the plurality of distributed platforms that perform distributed learning for a single learning task; a classification step for classifying the execution units of the plurality of distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism based on the metrics of the plurality of distributed platforms; and a scheduling step for scheduling the DP group to execute part of the single learning task using data parallelism, the PP group to execute part of the single learning task using pipeline parallelism, and the TP group to execute part of the single learning task using tensor parallelism.

[0017] According to one embodiment of the present invention, there is provided a scheduling method executed by a computer. The scheduling method may include an acquisition step of acquiring metrics of a plurality of distributed platforms from each of the distributed platforms that executes distributed learning for a single learning task. The scheduling method may include a classification step of classifying execution units of the plurality of distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism based on the metrics of the plurality of distributed platforms. The scheduling method may include a scheduling step of scheduling the DP group to execute a portion of the single learning task using data parallelism, the PP group to execute a portion of the single learning task using pipeline parallelism, and the TP group to execute a portion of the single learning task using tensor parallelism.

[0018] The above summary of the invention does not list all of the necessary features of the present invention, and subcombinations of these features may also constitute inventions.

[0019] FIG. 1 is a schematic diagram illustrating an example of a system 10. FIG. 2 is an explanatory diagram illustrating the extraction of one learning task 400 by a distribution infrastructure 200. FIG. 3 is an explanatory diagram illustrating the extraction of one learning task 400 by a distribution infrastructure 200. FIG. 4 is an explanatory diagram illustrating distributed learning in the system 10. FIG. 5 is a schematic diagram illustrating an example of the functional configuration of the distribution infrastructure 200. FIG. 6 is a schematic diagram illustrating an example of the functional configuration of the management infrastructure 100. FIG. 7 is a schematic diagram illustrating an example of the hardware configuration of a computer 1200 that functions as the management infrastructure 100 or the distribution infrastructure 200.

[0020] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0021] 1 schematically illustrates an example of a system 10. The system 10 includes a distributed infrastructure 200. The distributed infrastructure 200 may be an example of an execution infrastructure. The system 10 may include a plurality of distributed infrastructures 200. The system 10 may include a management infrastructure 100 that manages the plurality of distributed infrastructures 200. In the system 10 according to this embodiment, the management infrastructure 100 and the plurality of distributed infrastructures 200 may cooperate to control the RAN 310 and perform AI processing.

[0022] The RAN 310 may be a virtualized vRAN (Virtual RAN), and the system 10 may control the vRAN. The RAN 310 may be a physical RAN, and the system 10 may control the physical RAN. In this embodiment, a case where the RAN 310 is a vRAN will be mainly described as an example.

[0023] The AI ​​processing performed by the system 10 may include RAN-controlled AI processing (sometimes referred to as RAN_AI). The AI ​​processing performed by the system 10 may include non-RAN-controlled AI processing (sometimes referred to as non-RAN_AI).

[0024] The distributed infrastructure 200 may be a data center located in various locations. The distributed infrastructure 200 may be configured with multiple devices. The distributed infrastructure 200 may be realized on a virtualization platform made up of multiple devices. The distributed infrastructure 200 may also be realized by a single device. In other words, the distributed infrastructure 200 may be a distributed device.

[0025] The management infrastructure 100 may be a data center that manages multiple distributed infrastructures 200. The management infrastructure 100 may be configured by multiple devices. The management infrastructure 100 may be realized on a virtualization infrastructure made up of multiple devices. The management infrastructure 100 may also be realized by a single device. In other words, the management infrastructure 100 may be a management device.

[0026] The management infrastructure 100 may be referred to as a Core Brain, and the distributed infrastructure 200 may be referred to as a Regional Brain. While FIG. 1 illustrates an example in which a single-level management infrastructure 100 is arranged below the management infrastructure 100, this is not limiting. The distributed infrastructure 200 may have multiple levels. For example, when a two-level distributed infrastructure 200 is arranged below the management infrastructure 100, the management infrastructure 100 may be referred to as a Core Brain, the distributed infrastructure 200 at the level below that may be referred to as a Regional Brain, and the distributed infrastructure 200 at the level further below that may be referred to as a Sub-Regional Brain.

[0027] The distributed infrastructure 200 may be provided with one or more central processing units (CPUs). The distributed infrastructure 200 may be provided with one or more GPUs. The distributed infrastructure 200 may be provided with multiple super chips, each of which has a CPU and a GPU connected via an interconnect. The interconnect may have memory consistency and may be capable of achieving high bandwidth and low latency. In this way, the distributed infrastructure 200 may have CPU resources and GPU resources as computational resources.

[0028] 2 and 3 are explanatory diagrams for explaining the segmentation of one learning task 400 by the distribution platform 200. In the example shown in Figures 2 and 3, one learning task 400 is divided into two DPs, each of which is divided into four PPs, and each of which is divided into four TPs. The distribution platform 200 can segment the learning job in DP units 410, PP units 420, and TP units 430.

[0029] 4 is an explanatory diagram illustrating distributed learning in the system 10. The management platform 100 has a scheduler 102. The scheduler 102 may include a learning reception function for receiving learning requests, an architecture analysis function for analyzing the architecture of a learning model, a dataset analysis function for analyzing training teacher data 104, a scheduling function for scheduling learning using multiple distributed platforms 200, and a management function for managing learning status data 106.

[0030] The scheduler 102 receives metrics of the distributed infrastructure 200 from the distributed infrastructure 200. The scheduler 102 receives metrics from each of the multiple distributed infrastructures 200.

[0031] The metrics of the distributed infrastructure 200 may include the communication status between the distributed infrastructure 200 and other distributed infrastructures 200. The communication status may include throughput. The higher the throughput, the better the communication status. The communication status may include communication delay. The lower the communication delay, the better the communication status. The communication status may include communication quality. The higher the communication quality, the better the communication status. The communication status may include communication capacity. The higher the communication capacity, the better the communication status. The communication status may include communication bandwidth. The larger the communication bandwidth, the better the communication status. The communication status may include network cost. The lower the network cost, the better the communication status.

[0032] The metrics of the distributed infrastructure 200 may include the utilization status of the computational resources of the distributed infrastructure 200. The utilization status of the computational resources of the distributed infrastructure 200 may include the utilization status of the GPU resources of the distributed infrastructure 200.

[0033] The metrics of the distributed infrastructure 200 may include metrics of the execution unit of the distributed infrastructure 200. The execution unit of the distributed infrastructure 200 may include multiple execution entities. The execution entities may include a GPU instance 202. The GPU instance 202 includes multiple GPUs 204. The execution entities may include a GPU 204.

[0034] The metrics of the execution unit of the distributed infrastructure 200 may include metrics of the execution entities. The metrics of the execution entities may include the communication status between the execution entities. The metrics of the execution entities may include the communication status between the GPU instances 202. The metrics of the execution entities may include the communication status between the GPUs 204. The metrics of the execution entities may include the usage status of the computational resources of the execution entities. The metrics of the execution entities may include the usage status of the computational resources of the GPU instances 202. The metrics of the execution entities may include the usage status of the computational resources of the GPUs 204.

[0035] The scheduler 102 classifies the execution units of the multiple distributed infrastructures 200 into a DP group that uses DP, a PP group that uses PP, and a TP group that uses TP, based on metrics received from the multiple distributed infrastructures 200. For example, the scheduler 102 classifies the multiple execution entities of the multiple distributed infrastructures 200 into a DP group, a PP group, and a TP group, based on the metrics of each of the multiple execution entities of the multiple distributed infrastructures 200.

[0036] For example, the scheduler 102 includes, among the multiple execution entities of each of the multiple distributed platforms 200, multiple execution entities whose degree of goodness of their communication conditions with each other is greater than a predetermined first degree in the TP group, multiple execution entities whose degree of goodness of their communication conditions with each other is less than the first degree and greater than a second degree that is less than the first degree in the PP group, and multiple execution entities whose degree of goodness of their communication conditions with each other is less than the second degree in the DP group.

[0037] If the communication condition is throughput, the first degree and the second degree may be a threshold value for throughput. If the communication condition is communication delay, the first degree and the second degree may be a threshold value for communication delay. If the communication condition is communication quality, the first degree and the second degree may be a threshold value for communication quality. If the communication condition is communication capacity, the first degree and the second degree may be a threshold value for communication capacity. If the communication condition is communication bandwidth, the first degree and the second degree may be a threshold value for communication bandwidth. If the communication condition is network cost, the first degree and the second degree may be a threshold value for network cost. These threshold values ​​may be arbitrarily set, or may be changeable after being set.

[0038] The scheduler 102 may classify multiple execution entities into a DP group, a PP group, and a TP group based on the communication status between the execution entities and the usage status of the execution entities' computational resources. For example, the scheduler 102 may classify execution entities whose communication status is less than a second degree and whose remaining computational resources are less than a predetermined amount into a PP group instead of a DP group, among execution entities whose communication status is less than a first degree and greater than a second degree and whose remaining computational resources are less than a predetermined amount into a TP group instead of a PP group.

[0039] The scheduler 102 may exclude from the distributed learning any execution entity whose remaining computing resources are less than a predetermined amount, and classify the remaining execution entities into a DP group, a PP group, and a TP group based on the communication status between the execution entities.

[0040] As a specific example, as illustrated in Figure 4, the scheduler 102 classifies one GPU instance 202 of one distributed infrastructure 200 and one GPU instance 202 of another distributed infrastructure 200, whose communication conditions have a goodness degree less than the second degree, into a DP group; classifies one GPU instance 202 of one distributed infrastructure 200 and another GPU instance 202, whose communication conditions have a goodness degree less than the first degree and greater than the second degree, into a PP group; and classifies two GPUs 204 within one GPU instance 202 of one distributed infrastructure 200, whose communication conditions have a goodness degree greater than the first degree, into a TP group.

[0041] The scheduler 102 schedules the DP group to execute part of one learning task using the DP, the PP group to execute part of one learning task using the PP, and the TP group to execute part of one learning task using the TP. According to the classification results, the scheduler 102 divides the learning task into DP-based, PP-based, and DP-based learning jobs and assigns them to the DP group, PP group, and TP group. According to the assignment, the scheduler 102 distributes learning models and training data to each execution unit, and shares learning status data with each execution unit, so that the DP group can execute the DP, the PP group can execute the PP, and the TP group can execute the TP.

[0042] Figures 2 and 3 illustrate an example in which one learning task is divided into two DP-based learning jobs, each of the two DP-based learning jobs is divided into four PP-based learning jobs, and each of the four PP-based learning jobs is divided into four TP-based learning jobs. In the example shown in Figure 4, the scheduler 102 divides one learning task into two DP-based learning jobs, divides one of the two DP-based learning jobs into two PP-based learning jobs, and divides one of the two PP-based learning jobs into two TP-based learning jobs.

[0043] While FIG. 4 illustrates two distributed platforms 200, the number of distributed platforms 200 is not limited to this and may be greater. When multiple distributed platforms 200 exist and the distributed platforms 200 execute various processes, including RAN control, the communication status of multiple execution entities and the utilization status of computational resources change as appropriate. The scheduler 102 classifies and schedules each learning task by referring to the status of metrics for each epoch. This allows 3D parallelism learning to be performed in a manner that appropriately includes an increased number of execution units that can participate in learning, thereby achieving efficient utilization of the overall resources of the multiple distributed platforms 200. Furthermore, by switching to 3D parallelism learning using multiple execution units, excluding execution units that have difficulty continuing learning due to an increase in the load of other tasks, it is possible to prevent the occurrence of a situation in which the overall learning stagnates due to some execution units with increased load.

[0044] 5 shows an example of the functional configuration of the distribution infrastructure 200. The distribution infrastructure 200 includes a storage unit 212, an execution unit 214, a metrics generation unit 216, a metrics transmission unit 218, a training data acquisition unit 220, and a model acquisition unit 222.

[0045] The storage unit 212 stores various types of data. The storage unit 212 stores data used to control the RAN 310. The storage unit 212 stores data used to execute AI processing.

[0046] The execution unit 214 executes a RAN control function that controls the RAN 310 configured by a plurality of radio base stations 300. The execution unit 214 may execute control of a vRAN. The execution unit 214 controls the plurality of radio base stations 300 that configure the RAN 310 and cooperates with other distributed infrastructures 200 to control communications in the RAN 310.

[0047] The execution unit 214 executes AI processing. The execution unit 214 may execute RAN-controlled AI processing. The execution unit 214 may execute non-RAN-controlled AI processing.

[0048] The execution unit 214 executes distributed learning. The execution unit 214 executes a part of one learning task. The execution unit 214 may include multiple execution entities. The execution entity may include a GPU instance. The execution entity may include a GPU. Each of the multiple execution units included in the execution unit 214 may execute a part of one learning task.

[0049] The execution unit 214 executes the RAN control function and distributed learning using the computational resources of the distributed infrastructure 200. If there is a portion of the computational resources of the distributed infrastructure 200 that is not being used by the RAN control function, the execution unit 214 may execute distributed learning (i.e., a part of one learning task) using that portion. This allows control to be performed so that the RAN control function is prioritized, and then, if there are surplus resources, distributed learning is executed.

[0050] The metrics generation unit 216 generates metrics for the distributed infrastructure 200 (sometimes referred to as the local machine). The metrics generation unit 216 generates metrics for the communication status that changes due to changes in the communication environment of the local machine, and for the usage status of computational resources that changes when the execution unit 214 executes the RAN control function or AI processing.

[0051] The metrics generation unit 216 may generate metrics for its own machine in accordance with instructions from the management infrastructure 100. The metrics generation unit 216 generates metrics for its own machine for each epoch of one learning task in accordance with instructions from the management infrastructure 100, for example.

[0052] The metrics generator 216 may generate metrics for its own device according to a predetermined schedule. For example, the metrics generator 216 may generate metrics for its own device periodically according to a predetermined cycle.

[0053] The metrics of the distributed infrastructure 200 may include metrics of the execution unit of the distributed infrastructure 200. The metrics of the distributed infrastructure 200 may include metrics of each of the multiple execution units of the distributed infrastructure 200. The metrics may indicate the communication status between the multiple execution units of the distributed infrastructure 200 itself. The metrics may indicate the communication status between the multiple execution units of the distributed infrastructure 200 itself and other distributed infrastructures 200. The metrics may indicate the usage status of the computing resources of the execution entity of the distributed infrastructure 200 itself.

[0054] The metrics sending unit 218 sends the metrics generated by the metrics generating unit 216 to the management infrastructure 100. The metrics sending unit 218 may send the metrics to the management infrastructure 100 every time the metrics generating unit 216 generates them. The metrics sending unit 218 may send the metrics to the management infrastructure 100 for each epoch of one learning task.

[0055] The learning data acquisition unit 220 acquires learning data to be used in distributed learning. The learning data acquisition unit 220 may receive the learning data from the management platform 100. The model acquisition unit 222 acquires a learning model to be used in distributed learning.

[0056] The model acquisition unit 222 may receive a learning model from the management infrastructure 100. The model acquisition unit 222 may receive an entire learning model, a copy of an entire learning model, or a part of a learning model from the management infrastructure 100.

[0057] The execution unit 214 executes distributed learning using the learning data acquired by the learning data acquisition unit 220 and the learning model acquired by the model acquisition unit 222 .

[0058] 6 shows an example of the functional configuration of the management infrastructure 100. The management infrastructure 100 includes a storage unit 112, a learning data acquisition unit 114, a learning data analysis unit 116, a model management unit 118, a learning acceptance unit 120, a metrics acquisition unit 122, a classification unit 124, and a scheduling unit 126. It is not essential that the management infrastructure 100 include all of these units.

[0059] The learning data acquisition unit 114 acquires learning data for distributed learning. The learning data may be teacher data for learning. The learning data acquisition unit 114 may receive learning data from an external source. The learning data acquisition unit 114 may acquire learning data registered by a user who wishes to participate in distributed learning. The learning data acquisition unit 114 stores the acquired learning data in the storage unit 112.

[0060] The training data analysis unit 116 analyzes the training data acquired by the training data acquisition unit 114. The training data analysis unit 116 may analyze the training data and generate training data for distributed training from the training data. For example, the training data analysis unit 116 divides the training data for distributed training.

[0061] The model management unit 118 manages a learning model used in distributed learning. The model management unit 118 may generate a learning model. The model management unit 118 may replicate a learning model for DP. The model management unit 118 may divide a learning model for model parallel. The model management unit 118 may divide a learning model into multiple layer bundles for PP. The model management unit 118 may divide a tensor into multiple layers for TP.

[0062] The learning acceptance unit 120 accepts learning requests. The learning acceptance unit 120 may receive a learning request from a user who wishes to perform distributed learning from the communication terminal used by the user. The learning data acquisition unit 114 may acquire learning data to be used for the learning desired by the user, and the model management unit 118 may generate or acquire a learning model to be used for the learning desired by the user.

[0063] The metrics acquisition unit 122 receives metrics transmitted by the metrics transmission units 218 of each of the multiple distributed infrastructures 200 and stores them in the storage unit 112 .

[0064] The classification unit 124 classifies the execution units of the multiple distributed infrastructures 200 into a DP group, a PP group, and a TP group based on the metrics of the multiple distributed infrastructures 200. The classification unit 124 may classify the multiple execution entities of the multiple distributed infrastructures 200 into a DP group, a PP group, and a TP group based on the metrics of each of the multiple execution entities of the multiple distributed infrastructures 200.

[0065] For example, the classification unit 124 may include, among the multiple executing entities, multiple executing entities whose degree of goodness of their communication conditions with each other is greater than a predetermined first degree in a TP group, multiple executing entities whose degree of goodness of their communication conditions with each other is less than the first degree and greater than a second degree that is less than the first degree in a PP group, and multiple executing entities whose degree of goodness of their communication conditions with each other is less than the second degree in a DP group. The first degree and the second degree may be set arbitrarily and may be changeable after being set.

[0066] For example, the classification unit 124 classifies, among the multiple execution entities, multiple execution entities whose remaining amount of computing resources is less than a predetermined first amount into a TP group, multiple execution entities whose remaining amount of computing resources is more than the first amount and less than a second amount that is greater than the first amount into a PP group, and multiple execution entities whose remaining amount of computing resources is more than the second amount into a DP group. The first amount and the second amount may be set arbitrarily and may be changeable after being set.

[0067] The classification unit 124 may classify multiple execution entities into a DP group, a PP group, and a TP group based on the communication status between the execution entities and the usage status of the execution entities' computational resources.

[0068] For example, the classification unit 124 excludes from the distributed learning any of the multiple execution entities whose remaining amount of computational resources is less than a predetermined threshold, and includes, among the remaining multiple execution entities, multiple execution entities whose degree of goodness of their communication conditions with each other is greater than a predetermined first degree in the TP group, multiple execution entities whose degree of goodness of their communication conditions with each other is less than the first degree and greater than a second degree that is less than the first degree in the PP group, and multiple execution entities whose degree of goodness of their communication conditions with each other is less than the second degree in the DP group.

[0069] For example, the classification unit 124 includes, among multiple execution entities, multiple execution entities whose degree of goodness of their communication conditions with each other is greater than a predetermined first degree in the TP group; among multiple execution entities whose degree of goodness of their communication conditions with each other is less than the first degree and greater than a second degree which is less than the first degree, execution entities whose remaining amount of computing resources is less than a predetermined amount in the TP group, execution entities whose remaining amount of computing resources is less than a predetermined amount in the PP group; among multiple execution entities whose degree of goodness of their communication conditions with each other is less than the second degree, execution entities whose remaining amount of computing resources is less than a predetermined amount in the PP group, and execution entities whose remaining amount of computing resources is greater than a predetermined amount in the DP group.

[0070] The scheduling unit 126 schedules distributed learning by the multiple distribution infrastructures 200 in accordance with the classification by the classification unit 124. The scheduling unit 126 schedules distributed learning by the multiple execution entities of the multiple distribution infrastructures 200 in accordance with the classification by the classification unit 124.

[0071] The scheduling unit 126 schedules the DP group to execute a portion of a single learning task using the DP, the PP group to execute a portion of the single learning task using the PP, and the TP group to execute a portion of the single learning task using the TP. The scheduling unit 126 may cause the DP group to execute a learning job obtained by extracting a single learning task in DP units, the PP group to execute a learning job obtained by extracting a single learning task in PP units, and the TP group to execute a learning job obtained by extracting a single learning task in TP units. The scheduling unit 126 assigns learning data and learning models to each execution unit so that the DP group executes distributed learning using the DP, the PP group executes distributed learning using the PP, and the TP group executes distributed learning using the TP.

[0072] The scheduling unit 126 may assign the same learning model and divided learning data obtained by dividing learning data to each of the multiple execution units included in the DP group. The scheduling unit 126 may assign a bundle of layers obtained by dividing the learning model and respective learning data to each of the multiple execution units included in the PP group. The scheduling unit 126 may assign each of the multiple tensors to each of the multiple execution units included in the TP group.

[0073] The execution unit 214 of the distributed platform 200 performs distributed learning in accordance with instructions from the scheduling unit 126. Multiple execution units included in the execution unit 214 perform distributed learning in accordance with instructions from the scheduling unit 126. For example, if an execution unit itself is classified into a TP group, the execution unit performs TP distributed learning together with the other execution units included in the TP group using data assigned by the scheduling unit 126. For example, if an execution unit itself is classified into a PP group, the execution unit performs PP distributed learning together with the other execution units included in the PP group using data assigned by the scheduling unit 126. For example, if an execution unit itself is assigned to a DP group, the execution unit performs DP distributed learning together with the other execution units included in the DP group using data assigned by the scheduling unit 126.

[0074] The classification unit 124 may reclassify the distributed infrastructures 200 into DP groups, PP groups, and TP groups for each epoch of a learning task based on the metrics of the multiple distributed infrastructures 200, and the scheduling unit 126 may schedule the DP groups, PP groups, and TP groups to execute distributed learning for each epoch. For example, the metrics acquisition unit 122 may acquire metrics for the multiple distributed infrastructures 200 at the completion of one epoch of learning, and the classification unit 124 may classify the execution units of the multiple distributed infrastructures 200 into DP groups, PP groups, and TP groups based on the metrics of the multiple distributed infrastructures 200. The scheduling unit 126 may cause the newly classified DP groups to execute learning jobs extracted in DP units, the newly classified PP groups to execute learning jobs extracted in PP units, and the TP groups to execute learning jobs extracted in TP units. This means that, although it becomes necessary to extract learning jobs from the learning task according to the situation for each epoch, if the number of execution units or execution parts capable of performing distributed learning increases during the learning task, these execution units or execution parts can be immediately incorporated into the distributed learning, which contributes to optimizing the overall resource usage.

[0075] 7 schematically illustrates an example of the hardware configuration of a computer 1200 functioning as the management infrastructure 100 or the distribution infrastructure 200. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or can cause the computer 1200 to execute operations associated with the apparatus according to the present embodiment or one or more "parts," and / or can cause the computer 1200 to execute a process according to the present embodiment or steps of the process. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

[0076] The computer 1200 according to this embodiment includes a CPU 1212, a GPU 1213, a RAM 1214, and a graphics controller 1216, which are interconnected by a host controller 1210. The computer 1200 also includes input / output units such as a communications interface 1222, a storage device 1224, a DVD drive 1226, and an IC card drive, which are connected to the host controller 1210 via an input / output controller 1220. The DVD drive 1226 may be a DVD-ROM drive, a DVD-RAM drive, or the like. The storage device 1224 may be a hard disk drive, a solid-state drive, or the like. The computer 1200 also includes a ROM 1230 and legacy input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0077] The CPU 1212 operates according to programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into the graphics controller 1216 itself, and causes the image data to be displayed on the display device 1218.

[0078] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive 1226 reads programs or data from a DVD-ROM 1227 or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and / or writes programs and data to an IC card.

[0079] The ROM 1230 stores therein a boot program or the like that is executed by the computer 1200 upon activation, and / or programs that depend on the hardware of the computer 1200. The input / output chip 1240 may also connect various input / output units to the input / output controller 1220 via a USB port, a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0080] The programs are provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The programs are read from the computer-readable storage medium, installed in the storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by the CPU 1212. Information processing described in these programs is read by the computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or a method may be configured by implementing operations or processing of information in accordance with the use of the computer 1200.

[0081] For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, DVD-ROM 1227, or a recording medium such as an IC card, and transmits the read transmission data to a network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

[0082] Furthermore, the CPU 1212 may cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, the DVD drive 1226 (DVD-ROM 1227), an IC card, etc. to be read into the RAM 1214, and may perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

[0083] Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and may undergo information processing. The CPU 1212 may perform various types of processing on data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search / replacement, etc., as described throughout this disclosure and specified by the instruction sequences of the programs, and write the results back to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. on the recording medium. For example, if multiple entries, each having an attribute value of a first attribute associated with an attribute value of a second attribute, are stored on the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0084] The above-described programs or software modules may be stored in a computer-readable storage medium on or near the computer 1200. A recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can also be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

[0085] The blocks in the flowcharts and block diagrams in the present embodiments may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and / or a processor provided with computer-readable instructions stored on a computer-readable storage medium. The dedicated circuitry may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. The programmable circuitry may include reconfigurable hardware circuits, such as field programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements.

[0086] A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by an appropriate device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture, including instructions that can be executed to create means for performing the operations specified in the flowcharts or block diagrams. Examples of computer-readable storage media may include electronic, magnetic, optical, electromagnetic, and semiconductor storage media. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disc read-only memories (CD-ROMs), digital versatile discs (DVDs), Blu-ray discs, memory sticks, integrated circuit cards, and the like.

[0087] The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages ​​such as the “C” programming language or similar programming languages.

[0088] Computer-readable instructions may be provided locally or over a local area network (LAN), a wide area network (WAN) such as the Internet, to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, or programmable circuitry, such that the processor or programmable circuitry executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0089] Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0090] It should be noted that the execution order of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order.

[0091] 10 System, 100 Management infrastructure, 102 Scheduler, 104 Learning teacher data, 106 Learning status data, 112 Memory unit, 114 Learning data acquisition unit, 116 Learning data analysis unit, 118 Model management unit, 120 Learning acceptance unit, 122 Metrics acquisition unit, 124 Classification unit, 126 Scheduling unit, 200 Distribution infrastructure, 202 GPU instance, 204 GPU, 212 Memory unit, 214 Execution unit, 216 Metrics generation unit, 218 Metrics transmission unit, 220 Learning data acquisition unit, 222 Model acquisition unit, 300 Radio base station, 310 RAN, 400 Learning task, 410 DP unit, 420 PP unit, 430 TP unit, 1200 Computer, 1210 Host controller, 1212 CPU, 1213 GPU, 1214 RAM, 1216 graphics controller, 1218 display device, 1220 input / output controller, 1222 communication interface, 1224 storage device, 1226 DVD drive, 1227 DVD-ROM, 1230 ROM, 1240 input / output chip

Claims

1. A system comprising: a plurality of distributed platforms that perform distributed learning for a single learning task; and a management platform that manages the plurality of distributed platforms, wherein each of the plurality of distributed platforms has a metrics sending unit that sends metrics of the distributed platform to the management platform; and an execution unit that executes part of the single learning task, and the management platform has: a classification unit that classifies the execution units of the plurality of distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism based on the metrics of the plurality of distributed platforms; and a scheduling unit that schedules the DP group to execute part of the single learning task using data parallelism, the PP group to execute part of the single learning task using pipeline parallelism, and the TP group to execute part of the single learning task using tensor parallelism.

2. The system described in claim 1, wherein the execution unit includes a plurality of execution entities, the metrics sending unit sends metrics of each of the plurality of execution entities to the management platform, and the classification unit classifies the plurality of execution entities of the plurality of distributed platforms into the DP group, the PP group, and the TP group based on the metrics of each of the plurality of execution entities of the plurality of distributed platforms.

3. The system described in claim 2, wherein the multiple execution entities include multiple GPU instances and multiple GPUs, and the metrics sending unit sends metrics of the multiple GPUs and metrics of the multiple GPU instances to the management infrastructure.

4. The system described in claim 2, wherein the metrics indicate the communication status between the executing entities, and the classification unit includes, among the multiple executing entities, multiple executing entities whose degree of goodness of their communication status with each other is greater than a predetermined first degree in the TP group, multiple executing entities whose degree of goodness of their communication status with each other is less than the first degree and greater than a second degree that is less than the first degree in the PP group, and multiple executing entities whose degree of goodness of their communication status with each other is less than the second degree in the DP group.

5. The system described in claim 4, wherein the metrics further indicate the usage status of the execution entities' computing resources, and the classification unit classifies the multiple execution entities into the DP group, the PP group, and the TP group based on the communication status between the execution entities and the usage status of the execution entities' computing resources.

6. The system described in any one of claims 1 to 5, wherein the metrics sending unit sends metrics of the execution units of the distributed infrastructure to the management infrastructure for each epoch of the one learning task; the classification unit classifies the execution units of the multiple distributed infrastructures into the DP group, the PP group, and the TP group based on the metrics of the multiple distributed infrastructures for each epoch of the one learning task; and the scheduling unit schedules the DP group to execute a portion of the one learning task in data parallel, the PP group to execute a portion of the one learning task in pipeline parallel, and the TP group to execute a portion of the one learning task in tensor parallel for each epoch of the one learning task.

7. A system according to any one of claims 1 to 5, wherein the execution unit uses the computational resources of the distributed infrastructure to execute a RAN (Radio Access Network) control function that controls a RAN and a part of one of the learning tasks.

8. The system described in claim 7, wherein, if there is a portion of the computing resources of the distributed infrastructure that is not being used by the RAN control function, the execution unit uses that portion to execute part of the one learning task.

9. A management platform having: a metrics acquisition unit that acquires metrics of multiple distributed platforms from each of the multiple distributed platforms that perform distributed learning for a single learning task; a classification unit that classifies the execution units of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism based on the metrics of the multiple distributed platforms; and a scheduling unit that schedules the DP group to execute part of the single learning task using data parallelism, the PP group to execute part of the single learning task using pipeline parallelism, and the TP group to execute part of the single learning task using tensor parallelism.

10. A program for causing a computer to execute the following steps: an acquisition step for acquiring metrics of multiple distributed platforms from each of the multiple distributed platforms that perform distributed learning for a single learning task; a classification step for classifying the execution units of the multiple distributed platforms based on the metrics of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism; and a scheduling step for scheduling the DP group to execute part of the single learning task using data parallelism, the PP group to execute part of the single learning task using pipeline parallelism, and the TP group to execute part of the single learning task using tensor parallelism.

11. A scheduling method executed by a computer, comprising: an acquisition step of acquiring metrics of a plurality of distributed platforms from each of the plurality of distributed platforms that perform distributed learning for a single learning task; a classification step of classifying the execution units of the plurality of distributed platforms based on the metrics of the plurality of distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism; and a scheduling step of scheduling the DP group to execute part of the single learning task using data parallelism, the PP group to execute part of the single learning task using pipeline parallelism, and the TP group to execute part of the single learning task using tensor parallelism.

Images