System, management platform, program and scheduling method

Abstract

This invention provides a system comprising: multiple distributed platforms performing distributed learning on a learning task; and a management platform managing the multiple distributed platforms, each of the multiple distributed platforms having: a metric sending unit that sends the metrics of the distributed platform to a scheduler; and an execution unit that executes a portion of a learning task. The management platform has: a distributed learning decision unit that, based on the metrics of the multiple distributed platforms, determines a DP group using DP, a PP group using PP, and a TP group using TP among the multiple distributed platforms; and a scheduling unit that schedules the DP group to execute a portion of a learning task via DP, the PP group to execute a portion of a learning task via PP, and the TP group to execute a portion of a learning task via TP. The distributed platforms can perform RAN control and AI processing (including RAN control AI processing such as RIC (RAN Intelligent Controller) and non-RAN control AI processing).

Description

Technical Field

[0001] This invention relates to systems, management platforms, programs, and scheduling methods. Background Technology

[0002] Patent Document 1 describes an information processing apparatus comprising: a management unit that manages workers that can be used in distributed learning of a model; and a parameter server unit that issues tasks to the workers managed by the management unit, collects gradient data learned by the workers, and updates the model.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: International Publication No. 2021 / 220616 Summary of the Invention

[0006] The technical problem that the invention aims to solve

[0007] In typical distributed learning methods with teacher learning, there are data parallelism (sometimes abbreviated as DP) and model parallelism (sometimes abbreviated as MP). Model parallelism includes pipeline parallelism (sometimes abbreviated as PP) and tensor parallelism (sometimes abbreviated as TP). For example, in the case of distributed learning performed using multiple GPUs (Graphics Processing Units), in data parallelism, the dataset is partitioned according to the number of GPUs and assigned to each GPU, each GPU having a copy of the model, and each GPU learns its assigned dataset independently. Then, for example, after backpropagation of the error, synchronization is achieved by sharing the model's gradients among the GPUs. In pipeline parallelism, the model is partitioned into layers according to the number of GPUs, and the layers are assigned to each GPU for learning. In tensor parallelism, tensors are partitioned according to the number of GPUs and assigned to each GPU, each GPU processing its assigned tensor.

[0008] There is a trade-off between communication volume and memory efficiency in distributed learning (DP), partial learning (PP), and total learning (TP). Generally, the order TP, PP, DP requires more communication volume; that is, TP requires the most communication volume among DP, PP, and TP. Additionally, the order DP, PP, TP typically requires more total storage; that is, DP requires the most total storage across the entire distributed learning execution platform among TP, PP, and DP. The approach of three-dimensionally partitioning the learning task while effectively utilizing these elements is 3D parallel learning. Historically, to determine this 3D parallelization strategy, known GPU resources available for application in learning were either provided in advance or explored.

[0009] Solutions for solving technical problems

[0010] In the system described in this embodiment, dynamically changing DP, PP, and TP according to the situation helps to effectively utilize learning resources. As a specific example, the system described in this embodiment includes: a distributed platform having RAN control functions for controlling the RAN (Radio Area Network) and AI processing functions for performing AI (Artificial Intelligence) processing; and a management platform that manages multiple distributed platforms, which determines the DP group using DP, the PP group using PP, and the TP group using TP among the multiple distributed platforms based on metrics from the multiple distributed platforms.

[0011] As types of AI processing, there are AI processing associated with RAN control (sometimes referred to as RAN control AI processing) and AI processing not associated with RAN control (sometimes referred to as non-RAN control AI processing).

[0012] As an example of AI processing in RAN control, the RIC (RAN Intelligent Controller) is cited. RICs utilize AI to optimize RAN radio resources and automate RAN operations. RICs include Non-Real Time RICs (Non-RT RICs) and Near-Real Time RICs (Near-Real Time RICs). Non-RT RICs are sometimes referred to as Centralized RICs. Non-RT RICs reside within the SMO (Service Management and Orchestration) which manages and orchestrates the RAN. Non-RT RICs generate, notify, and send information related to RAN control to Near-RT RICs. For example, a Non-RT RIC might generate a RAN control-related learning model using machine learning data collected from the RAN and send it to a Near-RT RIC. Near-RT RICs are sometimes referred to as Distributed RICs. Compared to Non-RT RICs, Near-RT RICs are located near RAN nodes (RUs, DUs, and CUs) and perform RAN node control and resource control. Near-RT RICs perform more real-time processing compared to Non-RT RICs. For example, Near-RT RICs perform inference processing related to RAN control using models learned from Non-RT RICs. RAN control AI processing is not limited to RICs.

[0013] Non-RAN controlled AI processing can be equated to so-called MEC (Multi-access Edge Computing) applications. Examples of non-RAN controlled AI processing include surveillance AI execution processing that determines the situation within the shooting range of an input captured image, and response AI execution processing that outputs an answer to an input based on a user's query, but it is not limited to these.

[0014] In 3D parallel learning, optimizing this partitioning strategy enables optimal learning with speed as the primary objective. However, conventional methods rely on known instances used in the learning process; once learning based on this strategy begins, these instances cannot be released, making them unsuitable for use in distributed infrastructures with dynamically changing supply and demand, such as systems controlling RANs comprised of a management platform and multiple distributed platforms. In the system described in this embodiment, for example, a scheduler is introduced that can partition learning tasks into jobs at the round (or layer) level and schedule them to computing resources. By dynamically allocating jobs to distributed platforms during learning, low-cost learning utilizing the empty resources of distributed platforms can be achieved.

[0015] According to one embodiment of the present invention, a system is provided. The system may include multiple distributed platforms that perform distributed learning on a learning task. The system may include a management platform that manages the multiple distributed platforms. Each of the multiple distributed platforms may have a metric sending unit that sends the metrics of the distributed platform to a scheduler. Each of the multiple distributed platforms may have an execution unit that executes a portion of the learning task. The management platform may have a distributed learning decision unit that, based on the metrics of the multiple distributed platforms, determines which of the multiple distributed platforms uses data parallelism (DP), pipelined parallelism (PP), and tensor parallelism (TP). The management platform may have a scheduling unit that schedules the DP group to execute a portion of the learning task in data parallelism, the PP group to execute a portion of the learning task in pipelined parallelism, and the TP group to execute a portion of the learning task in tensor parallelism.

[0016] In the system, the execution unit may include multiple execution entities, and the measurement sending unit may send the respective measurements of the multiple execution entities to the management platform. The classification unit may classify the multiple execution entities on the multiple distributed platforms into the DP group, the PP group, and the TP group based on the respective measurements of the multiple execution entities on the multiple distributed platforms. The multiple execution entities may include multiple GPU instances and multiple GPUs, and the measurement sending unit may send the measurements of the multiple GPUs and the measurements of the multiple GPU instances to the management platform.

[0017] In any of the systems, the metric can represent the communication status between the execution entities. The classification unit can include execution entities whose mutual communication status is better than a predetermined first level in the TP group, execution entities whose mutual communication status is better than the first level but better than a second level smaller than the first level in the PP group, and execution entities whose mutual communication status is better than the second level in the DP group. The metric can also represent the utilization status of the execution entities' computing resources. The classification unit can classify the execution entities into the DP group, the PP group, and the TP group based on the communication status between the execution entities and the utilization status of the execution entities' computing resources.

[0018] In any of the systems, the metric sending unit can send the metrics of the execution units of the distributed platforms to the management platform for each round of the learning task. The classification unit can classify the execution units of the multiple distributed platforms into the DP group, the PP group, and the TP group based on the metrics of the multiple distributed platforms for each round of the learning task. The scheduling unit can schedule in the following manner: for each round of the learning task, the DP group executes a part of the learning task in data parallelism, the PP group executes a part of the learning task in pipeline parallelism, and the TP group executes a part of the learning task in tensor parallelism.

[0019] In either system, the execution unit can use the computing resources of the distributed platform to perform RAN control functions and a portion of the learning task. The execution unit can also use portions of the distributed platform's computing resources that are not currently used for the RAN control functions to perform a portion of the learning task.

[0020] According to one embodiment of the present invention, a management platform is provided. The management platform may include a metric acquisition unit that acquires metrics from multiple distributed platforms performing distributed learning on a learning task. The management platform may include a classification unit that, based on the metrics of the multiple distributed platforms, classifies the execution portions of the multiple distributed platforms into a DP group using data parallelism, a PP group using pipelined parallelism, and a TP group using tensor parallelism. The management platform may include a scheduling unit that schedules operations in such a manner that the DP group executes a portion of the learning task in data parallelism, the PP group executes a portion of the learning task in pipelined parallelism, and the TP group executes a portion of the learning task in tensor parallelism.

[0021] According to one embodiment of the present invention, a program is provided for causing a computer to perform the following steps: an acquisition step, acquiring metrics of a plurality of distributed platforms performing distributed learning on a learning task; a classification step, classifying the execution portions of the plurality of distributed platforms into a DP group using data parallelism, a PP group using pipelined parallelism, and a TP group using tensor parallelism based on the metrics of the plurality of distributed platforms; and a scheduling step, scheduling in such a way that the DP group executes a portion of the learning task in data parallelism, the PP group executes a portion of the learning task in pipelined parallelism, and the TP group executes a portion of the learning task in tensor parallelism.

[0022] According to one embodiment of the present invention, a scheduling method is provided, which is executed by a computer. The scheduling method may include an acquisition step, acquiring metrics of multiple distributed platforms performing distributed learning on a learning task. The scheduling method may include a classification step, classifying the execution portions of the multiple distributed platforms into groups using data parallelism (DP), pipelined parallelism (PP), and tensor parallelism (TP) based on the metrics of the multiple distributed platforms. The scheduling method may include a scheduling step, which schedules the DP group to execute a portion of the learning task in data parallelism, the PP group to execute a portion of the learning task in pipelined parallelism, and the TP group to execute a portion of the learning task in tensor parallelism.

[0023] Furthermore, the above summary of the invention does not list all the essential features of the invention. Additionally, sub-combinations of these feature groups can also constitute inventions. Attached Figure Description

[0024] Figure 1 This is a rough representation of an example of System 10.

[0025] Figure 2 This is an illustrative diagram used to explain the segmentation of a learning task 400 performed by the distributed platform 200.

[0026] Figure 3 This is an illustrative diagram used to explain the segmentation of a learning task 400 performed by the distributed platform 200.

[0027] Figure 4 This is an illustrative diagram used to explain distributed learning in System 10.

[0028] Figure 5 This is a simplified representation of the functional structure of the distributed platform 200.

[0029] Figure 6 This is a brief example of the functional structure of the management platform 100.

[0030] Figure 7 This is a brief illustration of an example of the hardware structure of a computer 1200 that functions as a management platform 100 or a distributed platform 200. Detailed Implementation

[0031] The present invention will now be described through embodiments thereof; however, these embodiments do not limit the invention as defined in the claims. Furthermore, the combinations of features described in the embodiments are not necessarily all necessary for the solutions provided by the invention.

[0032] Figure 1 This is a simplified representation of an example of system 10. System 10 includes a distributed platform 200. The distributed platform 200 can be an example of an execution platform. System 10 may include multiple distributed platforms 200. System 10 may include a management platform 100 that manages the multiple distributed platforms 200. In the system 10 according to this embodiment, the management platform 100 and the multiple distributed platforms 200 can collaboratively perform control and AI processing of RAN 310.

[0033] RAN310 can be a virtualized vRAN, and system 10 can perform vRAN control. RAN310 can also be a physical RAN, and system 10 can perform physical RAN control. In this embodiment, the example of RAN310 being a vRAN will be used for illustration.

[0034] The AI ​​processing performed by System 10 may include RAN control AI processing (sometimes referred to as RAN_AI). The AI ​​processing performed by System 10 may include non-RAN control AI processing (sometimes referred to as non-RAN_AI).

[0035] The distributed platform 200 can be a data center configured in various locations. The distributed platform 200 can be configured as multiple devices. The distributed platform 200 can be implemented on a virtualization platform based on multiple devices. The distributed platform 200 can also be implemented using a single device. That is, the distributed platform 200 can also be a distributed device.

[0036] Management platform 100 can be a data center that manages multiple distributed platforms 200. Management platform 100 can be composed of multiple devices. Management platform 100 can be implemented on a virtualization platform based on multiple devices. Management platform 100 can also be implemented by a single device. That is, management platform 100 can also be a management device.

[0037] Sometimes the management platform 100 is referred to as the Core Brain, and the distributed platform 200 is referred to as the Regional Brain. Furthermore, in... Figure 1 The example illustrates a scenario where a management platform 100 is configured with one layer below it, but this is not the only possibility. The distributed platform 200 can also have multiple layers. For instance, in the case where two distributed platforms 200 are configured below the management platform 100, the management platform 100 is sometimes referred to as the Core Brain, the next layer of distributed platforms 200 is called the Regional Brain, and the next layer of distributed platforms 200 is called the Sub-Regional Brain.

[0038] The distributed platform 200 can be configured with one or more CPUs (Central Processing Units). The distributed platform 200 can also be configured with one or more GPUs. Furthermore, the distributed platform 200 can be configured with multiple superchips connected to the CPUs and GPUs via interconnects. This interconnect can be a memory-coherent interconnect capable of achieving high frequency and low latency. Thus, the distributed platform 200 can have both CPU and GPU resources as computing resources.

[0039] Figure 2 , Figure 3 This is an illustrative diagram used to explain the segmentation of a learning task 400 performed by the distributed platform 200. Figure 2 , Figure 3 In the example shown, a learning task 400 is divided into two DPs, each of which is further divided into four PPs and four TPs. The distributed platform 200 can be sliced ​​into DP units 410, PP units 420, and TP units 430 as learning tasks.

[0040] Figure 4This is an explanatory diagram used to illustrate distributed learning in system 10. The management platform 100 has a scheduler 102. The scheduler 102 may have functions such as accepting learning requests, parsing the architecture of the learning model, parsing the dataset of teacher data 104 for learning, scheduling learning based on multiple distributed platforms 200, and managing the status data 106 in the learning process.

[0041] Scheduler 102 receives metrics from distributed platform 200. Scheduler 102 receives metrics from multiple distributed platforms 200 respectively.

[0042] Metrics for distributed platform 200 can include the communication status between distributed platform 200 and other distributed platforms 200. Communication status can include throughput. Higher throughput indicates better communication status. Communication status can include communication latency. Lower latency indicates better communication status. Communication status can include communication quality. Higher communication quality indicates better communication status. Communication status can include communication capacity. More communication capacity indicates better communication status. Communication status can include communication bandwidth. A larger bandwidth indicates better communication status. Communication status can include network cost. Lower network cost indicates better communication status.

[0043] The metrics for distributed platform 200 can include the utilization status of the computing resources of distributed platform 200. The utilization status of the computing resources of distributed platform 200 can include the utilization status of the GPU resources of distributed platform 200.

[0044] The metrics of the distributed platform 200 may include the metrics of the execution unit of the distributed platform 200. The execution unit of the distributed platform 200 may include multiple execution entities. An execution entity may include a GPU instance 202. A GPU instance 202 includes multiple GPUs 204. An execution entity may include GPUs 204.

[0045] The metrics for the execution units of the distributed platform 200 can include metrics for the execution entities. Metrics for the execution entities can include the communication status between the execution entities. Metrics for the execution entities can include the communication status between GPU instances 202. Metrics for the execution entities can include the communication status between GPUs 204. Metrics for the execution entities can include the utilization status of the computing resources of the execution entities. Metrics for the execution entities can include the utilization status of the computing resources of GPU instances 202. Metrics for the execution entities can include the utilization status of the computing resources of GPUs 204.

[0046] Scheduler 102 classifies the execution units of the multiple distributed platforms 200 into DP groups using DP, PP groups using PP, and TP groups using TP based on metrics received from the multiple distributed platforms 200. For example, scheduler 102 classifies multiple execution units of the multiple distributed platforms 200 into DP groups, PP groups, and TP groups based on the individual metrics of the multiple execution units of the multiple distributed platforms 200.

[0047] For example, scheduler 102 includes multiple execution entities of multiple distributed platforms 200 whose mutual communication status is better than a predetermined first level in the TP group, multiple execution entities whose mutual communication status is better than the first level but better than a second level smaller than the first level in the PP group, and multiple execution entities whose mutual communication status is better than the second level in the DP group.

[0048] When the communication condition is throughput, the first and second levels can be thresholds for throughput. When the communication condition is communication latency, the first and second levels can be thresholds for communication latency. When the communication condition is communication quality, the first and second levels can be thresholds for communication quality. When the communication condition is communication capacity, the first and second levels can be thresholds for communication capacity. When the communication condition is communication bandwidth, the first and second levels can be thresholds for communication bandwidth. When the communication condition is network cost, the first and second levels can be thresholds for network cost. These thresholds can be set arbitrarily and can be changed after being set.

[0049] Scheduler 102 can also classify multiple execution entities into DP, PP, and TP groups based on their communication status and the utilization of their computing resources. For example, scheduler 102 can classify execution entities with a communication quality level lower than the second level and less computing resource reserve than a predetermined amount into the PP group instead of the DP group. Similarly, scheduler 102 can classify execution entities with a communication quality level lower than the first level but higher than the second level and less computing resource reserve than a predetermined amount into the TP group instead of the PP group.

[0050] Scheduler 102 can exclude execution entities whose individual computing resource reserves are less than a predetermined amount from the distributed learning objects, and classify the remaining execution entities into DP group, PP group and TP group based on the communication status between the execution entities.

[0051] As a specific example, such as Figure 4As shown, scheduler 102 classifies one GPU instance 202 of one distributed platform 200 and one GPU instance 202 of another distributed platform 200, whose communication status is less than the second level, into the DP group; classifies one GPU instance 202 of one distributed platform 200 and one GPU instance 202 of another distributed platform 200, whose communication status is less than the first level but greater than the second level, into the PP group; and classifies two GPUs 204 of one GPU instance 202 of one distributed platform 200, whose communication status is greater than the first level, into the TP group.

[0052] Scheduler 102 schedules tasks in the following manner: DP group executes a portion of a learning task via DP, PP group executes a portion of a learning task via PP, and TP group executes a portion of a learning task via TP. Based on the classification results, scheduler 102 divides the learning task into learning jobs for DP units, PP units, and DP units, and assigns them to DP groups, PP groups, and TP groups. According to the assignment, scheduler 102 publishes the learning model and teacher data for each execution unit in a manner that enables DP groups to execute DP, PP groups to execute PP, and TP groups to execute TP, and shares learning state data with each execution unit.

[0053] exist Figure 2 , Figure 3 The example illustrates dividing a learning task into two DP units, further dividing each DP unit into four PP units, and finally dividing each PP unit into four TP units. Figure 4 In the example shown, scheduler 102 divides a learning task into two DP units of learning jobs, divides one of the two DP units of learning jobs into two PP units of learning jobs, and divides one of the two PP units of learning jobs into two TP units of learning jobs.

[0054] exist Figure 4The example illustrates two distributed platforms 200, but the number of distributed platforms 200 is not limited to this and can be more. Since multiple distributed platforms 200 exist and perform various processes, including RAN control, the communication status and computing resource utilization of the multiple execution entities will vary accordingly. For a learning task, the scheduler 102 classifies and schedules tasks based on the metric status within each round. Thus, as the number of execution units capable of participating in learning increases, 3D parallel learning that appropriately incorporates the execution units can be performed, and the overall resources of the multiple distributed platforms 200 can be effectively utilized. Furthermore, by switching to 3D parallel learning based on multiple execution units, excluding those that are struggling to continue learning due to increased load from other tasks, the occurrence of learning stagnation caused by increased load on a portion of the execution units can be suppressed.

[0055] Figure 5 Here is a simplified example of the functional structure of the distributed platform 200. The distributed platform 200 includes a storage unit 212, an execution unit 214, a metric generation unit 216, a metric transmission unit 218, a learning data acquisition unit 220, and a model acquisition unit 222.

[0056] Storage unit 212 stores various types of data. Storage unit 212 stores data used for controlling RAN310. Storage unit 212 stores data used for performing AI processing.

[0057] The execution unit 214 performs RAN control functions for the RAN 310, which is composed of multiple wireless base stations 300. The execution unit 214 can perform vRAN control. The execution unit 214 controls communication in the RAN 310 by controlling the multiple wireless base stations 300 that constitute the RAN 310 or by cooperating with other distributed platforms 200.

[0058] The execution unit 214 performs AI processing. The execution unit 214 can perform RAN control AI processing. The execution unit 214 can also perform non-RAN control AI processing.

[0059] Execution unit 214 performs distributed learning. Execution unit 214 executes a portion of a learning task. Execution unit 214 may include multiple execution entities. Execution entities may include GPU instances. Execution entities may include GPUs. Each of the multiple execution units included in execution unit 214 may also execute a portion of a learning task.

[0060] The execution unit 214 uses the computing resources of the distributed platform 200 to perform RAN control functions and distributed learning. If there are portions of the computing resources of the distributed platform 200 that are not used by the RAN control functions, the execution unit 214 can also use these portions to perform distributed learning (i.e., part of a learning task). Therefore, control can be performed in such a way that RAN control functions are performed while prioritizing them, and distributed learning is performed if there are remaining resources.

[0061] The measurement generation unit 216 generates measurements for the distributed platform 200 (sometimes referred to as the local machine). The measurement generation unit 216 generates measurements regarding changes in communication conditions due to changes in the local machine's communication environment, and changes in the application of computing resources due to the execution of RAN control functions and AI processing by the execution unit 214.

[0062] The metric generation unit 216 can generate local metrics according to instructions from the management platform 100. For example, the metric generation unit 216 generates local metrics for each round of a learning task according to instructions from the management platform 100.

[0063] The metric generation unit 216 can generate local metrics according to a predetermined schedule. For example, the metric generation unit 216 periodically generates local metrics according to a predetermined cycle.

[0064] The metrics of the distributed platform 200 may include metrics of the execution units of the distributed platform 200. The metrics of the distributed platform 200 may include individual metrics of multiple execution units of the distributed platform 200. Metrics may represent the communication status between multiple execution units of the local machine. Metrics may represent the communication status between the local machine and multiple execution units of other distributed platforms 200. Metrics may represent the utilization of computing resources by the local execution unit.

[0065] The metric sending unit 218 sends the metrics generated by the metric generation unit 216 to the management platform 100. The metric sending unit 218 can send the metrics to the management platform 100 each time a metric is generated by the metric generation unit 216. The metric sending unit 218 can also send the metrics to the management platform 100 for each round of a learning task.

[0066] The learning data acquisition unit 220 acquires learning data for distributed learning. The learning data acquisition unit 220 can receive learning data from the management platform 100. The model acquisition unit 222 acquires the learning model for distributed learning.

[0067] The model acquisition unit 222 can receive a learning model from the management platform 100. The model acquisition unit 222 can receive a complete learning model, a copy of a complete learning model, or a part of a learning model from the management platform 100.

[0068] The execution unit 214 performs distributed learning using the learning data acquired by the learning data acquisition unit 220 and the learning model acquired by the model acquisition unit 222.

[0069] Figure 6 This is a simplified example of the functional structure of the management platform 100. The management platform 100 includes a storage unit 112, a learning data acquisition unit 114, a learning data parsing unit 116, a model management unit 118, a learning processing unit 120, a metric acquisition unit 122, a classification unit 124, and a scheduling unit 126. The management platform 100 may not necessarily include all of the above.

[0070] The learning data acquisition unit 114 acquires learning data for distributed learning. The learning data may be teacher data for learning purposes. The learning data acquisition unit 114 can receive learning data from external sources. The learning data acquisition unit 114 can acquire learning data registered by users who wish to engage in distributed learning. The learning data acquisition unit 114 stores the acquired learning data in the storage unit 112.

[0071] The learning data parsing unit 116 parses the learning data acquired by the learning data acquisition unit 114. The learning data parsing unit 116 can parse the learning data and generate learning data for distributed learning based on it. For example, the learning data parsing unit 116 can divide the learning data into portions for distributed learning.

[0072] Model Management Unit 118 manages learning models used for distributed learning. Model Management Unit 118 can generate learning models. Model Management Unit 118 can replicate learning models for distributed learning (DP). Model Management Unit 118 can partition learning models for model parallelization. Model Management Unit 118 can partition learning models into multiple layers for distributed learning (PP). Model Management Unit 118 can partition tensors into multiple layers for distributed learning (TP).

[0073] The learning processing unit 120 accepts learning requests. The learning processing unit 120 can receive learning requests from users who wish to perform distributed learning from the communication terminal of the user application. The learning data acquisition unit 114 can acquire learning data for the learning requested by the user, and the model management unit 118 can generate or acquire a learning model for the learning requested by the user.

[0074] The measurement acquisition unit 122 receives the measurements sent by the respective measurement sending units 218 of the multiple distributed platforms 200 and stores them in the storage unit 112.

[0075] The classification unit 124 classifies the execution units of the multiple distributed platforms 200 into DP, PP, and TP groups based on the metrics of the multiple distributed platforms 200. The classification unit 124 can classify multiple execution entities of the multiple distributed platforms 200 into DP, PP, and TP groups based on the individual metrics of the multiple execution entities of the multiple distributed platforms 200.

[0076] For example, classification unit 124 includes multiple execution entities whose mutual communication quality is greater than a predetermined first level in the TP group, multiple execution entities whose mutual communication quality is less than the first level but greater than a second level lower than the first level in the PP group, and multiple execution entities whose mutual communication quality is less than the second level in the DP group. The first level and the second level can be arbitrarily set or changed after being set.

[0077] For example, classification unit 124 includes multiple execution entities whose remaining computing resources are less than a predetermined first amount in the TP group, multiple execution entities whose remaining computing resources are more than the first amount but less than a second amount greater than the first amount in the PP group, and multiple execution entities whose remaining computing resources are more than the second amount in the DP group. The first amount and the second amount can be set arbitrarily or changed after being set.

[0078] The classification unit 124 can also classify multiple execution entities into DP group, PP group and TP group based on the communication status between the execution entities and the application status of the computing resources of the execution entities.

[0079] For example, classification unit 124 excludes execution subjects whose remaining computing resources are less than a predetermined threshold from the distributed learning objects, includes execution subjects whose mutual communication is better than a predetermined first level in the TP group, includes execution subjects whose mutual communication is better than the first level but better than a second level smaller than the first level in the PP group, and includes execution subjects whose mutual communication is better than the second level in the DP group.

[0080] For example, classification unit 124 includes multiple execution entities whose mutual communication quality is greater than a predetermined first level in the TP group; multiple execution entities whose mutual communication quality is less than the first level but greater than a second level smaller than the first level, and whose remaining computing resources are less than a predetermined amount in the TP group; execution entities whose remaining computing resources are less than a predetermined amount in the PP group; execution entities whose mutual communication quality is less than the second level, and whose remaining computing resources are less than a predetermined amount in the PP group; and execution entities whose remaining computing resources are greater than a predetermined amount in the DP group.

[0081] The scheduling unit 126 schedules distributed learning based on the classification of the classification unit 124, and schedules distributed learning of multiple execution entities based on the multiple distributed platforms 200.

[0082] The scheduling unit 126 performs scheduling in the following manner: the DP group executes a portion of a learning task through DP, the PP group executes a portion of a learning task through PP, and the TP group executes a portion of a learning task through TP. The scheduling unit 126 can cause the DP group to execute learning tasks sliced ​​from a learning task on a DP basis, the PP group to execute learning tasks sliced ​​from a learning task on a PP basis, and the TP group to execute learning tasks sliced ​​from a learning task on a TP basis. To enable the DP group to execute DP-based distributed learning, the PP group to execute PP-based distributed learning, and the TP group to execute TP-based distributed learning, the scheduling unit 126 allocates learning data and learning models to their respective execution units.

[0083] The scheduling unit 126 can allocate the same learning model and the segmented learning data obtained from the segmented learning data to the multiple execution units contained in the DP group. The scheduling unit 126 can allocate the layer groups obtained from the segmented learning model and their respective learning data to the multiple execution units contained in the PP group. The scheduling unit 126 can allocate multiple tensors to each of the multiple execution units contained in the TP group.

[0084] The execution unit 214 of the distributed platform 200 performs distributed learning according to instructions from the scheduling unit 126. Multiple execution units included in the execution unit 214 perform distributed learning according to instructions from the scheduling unit 126. For example, if it is classified as a TP group, the execution unit performs TP distributed learning using data allocated from the scheduling unit 126 along with other execution units included in the TP group. Similarly, if it is classified as a PP group, the execution unit performs PP distributed learning using data allocated from the scheduling unit 126 along with other execution units included in the PP group. And if it is assigned to a DP group, the execution unit performs DP distributed learning using data allocated from the scheduling unit 126 along with other execution units included in the DP group.

[0085] The classification unit 124 can also schedule tasks in the following manner: Based on the metrics of multiple distributed platforms 200, the DP group, PP group, and TP group are reclassified for each round of a learning task. The scheduling unit 126 then schedules the DP group, PP group, and TP group to perform distributed learning for each round. For example, the metric acquisition unit 122 acquires the metrics of multiple distributed platforms 200 at the completion of a learning round. Based on these metrics, the classification unit 124 classifies the execution units of the multiple distributed platforms 200 into DP group, PP group, and TP group. The scheduling unit 126 can then schedule the newly classified DP group to perform learning tasks segmented in DP units, the newly classified PP group to perform learning tasks segmented in PP units, and the TP group to perform learning tasks segmented in TP units. Thus, although it is necessary to segment learning tasks from the corresponding learning tasks for each round, if an execution unit or execution unit capable of performing distributed learning is added midway through a learning task, it can be immediately incorporated into the distributed learning, which helps to optimize the overall application resources.

[0086] Figure 7 This section outlines an example of the hardware structure of a computer 1200 that functions as a management platform 100 or a distributed platform 200. A program installed on the computer 1200 enables it to function as one or more "parts" of the apparatus according to this embodiment, or to perform operations associated with the apparatus or those one or more "parts," and / or to perform processes or steps of those processes according to this embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to perform specific operations associated with several or all of the blocks in the flowcharts and block diagrams described in this specification.

[0087] The computer 1200 based on this embodiment includes a CPU 1212, a GPU 1213, a RAM 1214, and a graphics controller 1216, which are interconnected via a main controller 1210. The computer 1200 also includes input / output units such as a communication interface 1222, a storage device 1224, a DVD drive 1226, and an IC card driver, which are connected to the main controller 1210 via an input / output controller 1220. The DVD drive 1226 can be a DVD-ROM drive or a DVD-RAM drive, etc. The storage device 1224 can be a hard disk drive or a solid-state drive, etc. The computer 1200 also includes a ROM 1230 and conventional input / output units such as a keyboard, which are connected to the input / output controller 1220 via an input / output chip 1240.

[0088] CPU 1212 operates according to the program stored in ROM 1230 and RAM 1214, thereby controlling each unit. Graphics controller 1216 obtains image data generated by CPU 1212 from frame buffers or other sources provided in RAM 1214 or from itself, and displays the image data on display device 1218.

[0089] The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 within the computer 1200. The DVD drive 1226 reads programs or data from the DVD-ROM 1227, etc., and provides them to the storage device 1224. The IC card driver reads programs and data from the IC card and / or writes programs and data to the IC card.

[0090] ROM 1230 stores the boot program executed by computer 1200 during activation, and / or programs that depend on the hardware of computer 1200. Input / output chip 1240 can also connect various input / output units to input / output controller 1220 via USB port, parallel port, serial port, keyboard port, mouse port, etc.

[0091] The program is provided by a computer-readable storage medium such as a DVD-ROM 1227 or an IC card. The program is read from the computer-readable storage medium and installed in a storage device 1224, RAM 1214, or ROM 1230, which is also an example of a computer-readable storage medium, and is executed by the CPU 1212. The information processing described within this program is read by the computer 1200, resulting in cooperation between the program and the aforementioned types of hardware resources. The apparatus or method can be configured to perform information manipulation or processing according to the use of the computer 1200.

[0092] For example, when communication is performed between computer 1200 and an external device, CPU 1212 can execute a communication program loaded into RAM 1214, and based on the processing described in the communication program, command communication processing is performed on communication interface 1222. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in a recording medium such as RAM 1214, storage device 1224, DVD-ROM 1227, or IC card, and sends the read transmission data to the network, or writes received data received from the network into a receive buffer area provided on the recording medium, etc.

[0093] In addition, the CPU 1212 can enable the RAM 1214 to read all or necessary portions of files or databases stored in external recording media such as storage device 1224, DVD drive 1226 (DVD-ROM 1227), IC card, etc., and perform various types of processing on the data in the RAM 1214. Next, the CPU 1212 can write the processed data back to the external recording media.

[0094] Various types of information, such as programs, data, tables, and databases, can be stored in the recording medium and can be processed. The CPU 1212 can perform various types of processing on data read from RAM 1214, including operations, information processing, conditional judgments, conditional branches, unconditional branches, information retrieval / replacement, etc., specified by a sequence of program commands as described in various parts of this disclosure, and write the results back to RAM 1214. Furthermore, the CPU 1212 can retrieve information from files, databases, etc., within the recording medium. For example, if the recording medium contains multiple entries, each containing an attribute value of a first attribute associated with a second attribute value, the CPU 1212 can retrieve from these multiple entries an entry whose first attribute value matches a specified condition, and read the attribute value of the second attribute stored in that entry, thereby obtaining the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0095] The programs or software modules described above can be stored on or near the computer 1200 in a computer-readable storage medium. Alternatively, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the program to the computer 1200 via the network.

[0096] In this embodiment, the flowcharts and blocks in the block diagrams can represent steps of an operation to be performed or "parts" of a device that performs the operation. Specific steps and "parts" can be installed via dedicated circuitry, programmable circuitry supplied along with computer-readable commands stored on a computer-readable storage medium, and / or a processor supplied along with computer-readable commands stored on a computer-readable storage medium. Dedicated circuitry can include digital and / or analog hardware circuitry, and can also include integrated circuits (ICs) and / or discrete circuitry. Programmable circuitry can include reconfigurable hardware circuitry such as field-programmable gate arrays (FPGAs) and programmable logic arrays (PLAs), which include logic products, logical sums, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and storage elements.

[0097] Computer-readable storage media can include any tangible device capable of storing commands that can be executed by an appropriate device. As a result, a computer-readable storage medium having commands stored therein comprises commands that can be executed to create a unit for performing operations specified in a flowchart or block diagram. Examples of computer-readable storage media include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, etc. More specific examples of computer-readable storage media include floppy disks (registered trademark), magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), Blu-ray disc (registered trademark), memory stick, integrated circuit cards, etc.

[0098] Computer-readable commands may include assembly commands, instruction set architecture (ISA) commands, machine commands, machine-dependent commands, microcode, firmware commands, status setting data, or any combination of source code or object code written in an object-oriented programming language such as Smalltalk (registered trademark), JAVA (registered trademark), C++, or a programming language such as "C" or similar languages ​​that include conventional procedural programming languages.

[0099] Computer-readable commands can be provided locally or via a wide area network (WAN) such as a local area network (LAN) or the Internet to the processor or programmable circuitry of a general-purpose computer, special-purpose computer, or other programmable data processing device, so that the processor or programmable circuitry of the general-purpose computer, special-purpose computer, or other programmable data processing device executes the computer-readable commands to generate units for performing the operations specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

[0100] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. Those skilled in the art will naturally be able to make various modifications or improvements to the above embodiments. As can be seen from the claims, such modifications or improvements are also included within the technical scope of the present invention.

[0101] It should be noted that the execution order of actions, sequences, steps, and stages in the apparatus, system, program, and method shown in the claims, specification, and drawings can be implemented in any order, unless specifically stated as "before" or "in advance," and unless the output of a preceding process is used in a subsequent process. Even if terms such as "firstly" or "next" are used to describe the flow of actions in the claims, specification, and drawings for convenience, this does not imply that the actions must be performed in that order.

[0102] Explanation of reference numerals in the attached figures

[0103] 10. System; 100. Management Platform; 102. Scheduler; 104. Teacher Data for Learning; 106. Learning Status Data; 112. Storage Department; 114. Learning Data Acquisition Department; 116. Learning Data Parsing Department; 118. Model Management Department; 120. Learning Processing Department; 122. Measurement Acquisition Department; 124. Classification Department; 126. Scheduling Department; 200. Distributed Platform; 202. GPU Instance; 204. GPU; 212. Storage Department; 214. Execution Department; 216. Measurement Generation Department; 218. Measurement Sending Department; 220. Learning Data Acquisition Department; 222. Model Type acquisition unit; 300, wireless base station; 310, RAN; 400, learning task; 410, DP unit; 420, PP unit; 430, TP unit; 1200, computer; 1210, host controller; 1212, CPU; 1213, GPU; 1214, RAM; 1216, graphics controller; 1218, display device; 1220, input / output controller; 1222, communication interface; 1224, storage device; 1226, DVD drive; 1227, DVD-ROM; 1230, ROM; 1240, input / output chip.

Claims

1. A system that possesses: Multiple distributed platforms perform distributed learning on a single learning task; and The management platform manages the multiple distributed platforms. Each of the plurality of distributed platforms has: The measurement sending unit sends the measurements of the distributed platform to the management platform; as well as The execution department performs part of the learning task. The management platform has the following features: The classification department, based on the metrics of the multiple distributed platforms, classifies the execution parts of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism. as well as The scheduling department schedules tasks in the following manner: the DP group executes a portion of the learning task in parallel via data, the PP group executes a portion of the learning task in parallel via pipeline, and the TP group executes a portion of the learning task in parallel via tensors.

2. The system according to claim 1, wherein, The execution unit includes multiple execution entities. The measurement sending unit sends the respective measurements of the multiple execution entities to the management platform. The classification department classifies the multiple execution entities of the multiple distributed platforms into the DP group, the PP group, and the TP group based on the respective metrics of the multiple execution entities of the multiple distributed platforms.

3. The system according to claim 2, wherein, The multiple execution entities include multiple GPU instances and multiple GPUs. The metric sending unit sends the metrics of the multiple GPUs and the metrics of the multiple GPU instances to the management platform.

4. The system according to claim 2, wherein, The metric represents the communication status between the executing entities. The classification department includes multiple execution entities whose mutual communication quality is greater than a predetermined first level in the TP group, multiple execution entities whose mutual communication quality is less than the first level but greater than a second level smaller than the first level in the PP group, and multiple execution entities whose mutual communication quality is less than the second level in the DP group.

5. The system according to claim 4, wherein, The metric also represents the utilization status of the computing resources of the executing entity. The classification department categorizes the multiple execution entities into the DP group, the PP group, and the TP group based on their communication status with each other and their application of computing resources.

6. The system according to any one of claims 1 to 5, wherein, The metric sending unit sends the metrics of the execution unit of the distributed platform to the management platform for each round of the learning task. The classification department, based on the metrics of the multiple distributed platforms, classifies the execution parts of the multiple distributed platforms into the DP group, the PP group, and the TP group for each round of the learning task. The scheduling unit performs scheduling in the following manner: for each round of the learning task, the DP group executes a portion of the learning task in parallel via data, the PP group executes a portion of the learning task in parallel via pipeline, and the TP group executes a portion of the learning task in parallel via tensor.

7. The system according to any one of claims 1 to 5, wherein, The execution unit uses the computing resources of the distributed platform to perform RAN control functions for controlling the RAN and as part of the learning task.

8. The system according to claim 7, wherein, When there are portions of the computing resources on the distributed platform that are not used by the RAN control function, the execution unit uses these portions to perform a part of the learning task.

9. A management platform, comprising: The metric acquisition unit acquires the metrics of the distributed platforms from multiple distributed platforms that perform distributed learning on a learning task. The classification department, based on the metrics of the multiple distributed platforms, classifies the execution parts of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism. as well as The scheduling department schedules the learning task in the following ways: the DP group executes a portion of the learning task in parallel via data, the PP group executes a portion of the learning task in parallel via pipeline, and the TP group executes a portion of the learning task in parallel via tensor.

10. A program for causing a computer to perform the following steps: The acquisition step involves obtaining the metrics of each of the multiple distributed platforms that perform distributed learning on a learning task. The classification step, based on the metrics of the multiple distributed platforms, classifies the execution portions of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism. as well as The scheduling steps are performed as follows: the DP group executes a portion of the learning task in parallel via data, the PP group executes a portion of the learning task in parallel via pipeline, and the TP group executes a portion of the learning task in parallel via tensor.

11. A scheduling method executed by a computer, the scheduling method comprising, The acquisition step involves obtaining the metrics of each of the multiple distributed platforms that perform distributed learning on a learning task. The classification step, based on the metrics of the multiple distributed platforms, classifies the execution portions of the multiple distributed platforms into a DP group that uses data parallelism, a PP group that uses pipeline parallelism, and a TP group that uses tensor parallelism. as well as The scheduling steps are performed as follows: the DP group executes a portion of the learning task in data parallelism, the PP group executes a portion of the learning task in pipeline parallelism, and the TP group executes a portion of the learning task in tensor parallelism.

Images