A heterogeneous parallel computing system and distributed training method

Abstract

The present application discloses a heterogeneous parallel computing system and a distributed training method, which belongs to the field of computers, including a task flow management module for parsing model distributed training tasks, defining corresponding nodes, including task management nodes, parameter service nodes, and heterogeneous computing nodes, decomposing the training tasks into multiple subtasks, and assigning these subtasks to different nodes for parallel computing; a heterogeneous cluster resource management module for managing and calling hardware resources of various structures to form nodes, monitoring and guaranteeing hardware resources; a data distribution management module for managing and processing data information in the system; and a parallel strategy control module for completing heterogeneous distributed computing and model convergence. This heterogeneous parallel computing system supports high scalability, and can split and assign a deep learning heterogeneous task to intelligent computing hardware of different resource types, execute tasks concurrently, and improve computing performance and efficiency.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a heterogeneous parallel computing system and a distributed training method. Background Art

[0002] In the development of artificial intelligence and big models, many hardware resources designed for artificial intelligence have emerged, such as GPU, TPU and NPU. These hardware resources play an important role in providing high-performance computing capabilities. However, with the increasing demand for big model training, a single computing resource can no longer meet the requirements, and in most companies, different types of hardware resources also vary, resulting in low resource utilization.

[0003] The existing technologies mainly include traditional task scheduling systems and distributed computing frameworks. Traditional task scheduling systems mainly use job scheduling algorithms to schedule tasks, but they are not friendly to the scheduling of heterogeneous hardware resources and cannot fully utilize the performance advantages of different hardware. Distributed computing frameworks such as Hadoop and Spark can perform distributed task scheduling, but their support for the scheduling of heterogeneous hardware resources is limited.

[0004] Traditional computing frameworks and AI development platforms that support parallel computing mostly use homogeneous AI computing cards at the hardware resource level, such as GPU-based platforms or NPU-based platforms, which cannot fully utilize the hardware resources of different types of AI computing cards, resulting in low resource utilization; the deployment of some AI development platforms also uses heterogeneous intelligent computing resources, but only involves resource heterogeneity, labels different resources, and matches tasks with corresponding resources and computing frameworks according to business needs. The calculation execution of a specific single task is still based on homogeneous computing power and cannot cross heterogeneous computing power. It is not task-level heterogeneity and cannot be unified across multiple frameworks and multiple hardware. There is a lack of adaptive resource scheduling mechanism, and it is impossible to flexibly schedule resources according to actual needs.

[0005] Therefore, there is an urgent need for a heterogeneous parallel computing system that can solve the above problems. Summary of the invention

[0006] In order to address the deficiencies of the prior art, this application provides a heterogeneous parallel computing framework and usage method, which implements resource monitoring, cluster creation, job scheduling and other functions for heterogeneous AI computing cards such as GPU and NPU, and enables the same computing task to call multiple heterogeneous AI computing cards for parallel computing, thus realizing task-level heterogeneous computing. By encapsulating the training code of a single machine model and cooperating with the creation of a heterogeneous computing power environment, heterogeneous parallel computing of deep learning models in a heterogeneous AI computing power environment can be realized.

[0007] The technical effects to be achieved by this application are achieved through the following solutions:

[0008] According to a first aspect of the present application, a heterogeneous parallel computing system is provided, including a task flow management module, a heterogeneous cluster resource management module, a data distribution management module and a parallel strategy control module, wherein:

[0009] The task flow management module is used to parse the model distributed training task, define the corresponding nodes, including the task management node, the parameter service node and the heterogeneous computing node, decompose the training task into multiple subtasks, and assign these subtasks to different nodes for parallel computing;

[0010] The heterogeneous cluster resource management module is used to manage and call hardware resources of various structures to form the nodes, and monitor and ensure the hardware resources;

[0011] The data distribution management module is used to manage and process the data information in the system, including at least pre-processing, segmentation, distribution management and polling of the data information;

[0012] The parallel strategy control module is used to complete heterogeneous distributed computing and model convergence.

[0013] Preferably, the task flow management module allocates the training task to the node using uniform scheduling or resource status scheduling according to the priority, dependency and resource requirements of the training task; wherein:

[0014] The uniform scheduling is to evenly schedule all the nodes that own the resource for calculation;

[0015] The resource status scheduling is to allocate the resources of each node in descending order to the nodes that meet the training task.

[0016] Preferably, the multiple hardware resources with different structures include but are not limited to CPU, GPU, NPU, TPU and BPU. The hardware resources are divided into multiple resource groups for management. The same hardware resources are allocated in the same resource group. There are several resource groups for the same hardware resources, and each resource group forms a node.

[0017] Preferably, a heterogeneous distributed cluster is constructed between the nodes through IP and port. The first-level task scheduling uses gRPC communication to send computing tasks to each resource group. The second-level task scheduling uses a high-speed communication channel and uses the ALLReduce method for calculation; and monitors all nodes and connected hardware resources in the cluster. When a node or a hardware resource fails, an alarm, repair or migration can be performed.

[0018] Preferably, the data distribution management module preprocesses the data information including: performing size transformation, normalization, data enhancement and word segmentation on the data information;

[0019] The splitting process includes average splitting or weighted splitting of data information, where weighted splitting is based on the proportion of each node's total computing power to all nodes;

[0020] The distribution management is to split all the data information generated by the training task and send it to each node.

[0021] Preferably, after the data information on all heterogeneous computing nodes completes one iteration of training, the parallel strategy control module allows each heterogeneous computing node to randomly transfer a certain proportion of data to an adjacent heterogeneous computing node.

[0022] According to a second aspect of the present application, a distributed training method using the above-mentioned heterogeneous parallel computing system is provided, comprising the following steps:

[0023] Step 1: Create a distributed training task and submit it to a heterogeneous parallel computing system;

[0024] Step 2: The task flow management module parses the distributed training task, defines the corresponding task management nodes, parameter service nodes and heterogeneous computing nodes, and sends the resource demand information to the heterogeneous cluster resource management module;

[0025] Step 3: The heterogeneous cluster resource management module creates corresponding task management nodes, parameter service nodes, and heterogeneous computing nodes according to the analyzed task resource requirements;

[0026] Step 4: The task flow management module performs distributed training task scheduling: the task management node divides the distributed training tasks according to the heterogeneous computing nodes and sends them to each resource group; at the same time, the data segmentation strategy information is sent to the data distribution management module, and the parallel training strategy information is sent to the parallel strategy control module;

[0027] Step 5: The data distribution management module preprocesses, splits and distributes the corresponding data sets of the distributed training task;

[0028] Step 6: The parallel strategy control module performs corresponding distributed training;

[0029] Step 7: The task process management module monitors the distributed training tasks in execution, including task status monitoring and training log monitoring;

[0030] Step 8: After the distributed training task is completed, the task process management module notifies the heterogeneous cluster resource management module to release the task management node, parameter service node and heterogeneous computing node corresponding to the distributed training task.

[0031] Preferably, in step 1, it specifically includes:

[0032] Step 11: Create a model development project as needed and record relevant project information;

[0033] Step 12: Define the relevant model training tasks, write project code using a general computing framework, and define the model, data set, optimizer, and loss function required for stand-alone operation;

[0034] Step 13: Configure the computing resources, data segmentation strategy, and parallel training strategy required for distributed training for the model training task;

[0035] Step 14: Submit the model development task to the heterogeneous parallel computing system.

[0036] Preferably, in step 4, the task management node divides the distributed training tasks according to heterogeneous computing nodes, and sends the first-level tasks to each resource group. Each resource group divides the first-level tasks into several second-level tasks, and distributes the second-level tasks to all hardware resources in the resource group.

[0037] Preferably, in step 6, while the data distribution management module sends the model training data, all heterogeneous computing nodes start synchronizing the initial global model information of the parameter service node;

[0038] Based on the parallel training strategy information parsed by the task flow management module and the scheduling of distributed training tasks, heterogeneous computing nodes perform local model training and interact with parameter service nodes for model information to jointly complete the execution of model distributed training tasks.

[0039] According to one embodiment of the present application, the beneficial effects of adopting the heterogeneous parallel computing system of the present application are: the heterogeneous parallel computing system supports high scalability, supports multiple general computing frameworks, and supports multiple types of intelligent computing hardware devices;

[0040] Ability to split a deep learning heterogeneous task and distribute it to intelligent computing hardware of different resource types, execute tasks concurrently, and improve computing performance and efficiency;

[0041] Heterogeneous parallel computing systems can make full use of intelligent computing hardware of different resource types and adapt to more AI model development needs. They can use homogeneous AI computing cards to execute distributed training of models as needed, or they can use heterogeneous AI computing cards to execute distributed training of models. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the existing technical solutions, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 This is a flow chart of a distributed training method using a heterogeneous parallel computing system in one embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0045] The heterogeneous parallel computing system in one embodiment of the present application includes a task flow management module, a heterogeneous cluster resource management module, a data distribution management module and a parallel strategy control module, wherein:

[0046] The task process management module supports encapsulation and conversion of original codes into heterogeneous distributed training scripts, thereby realizing the business processes and functions of distributed parallel training on heterogeneous AI computing nodes;

[0047] It is used to parse model distributed training tasks, define corresponding task management nodes, parameter service nodes and heterogeneous computing nodes, decompose the training tasks into multiple subtasks, and assign these subtasks to different nodes for parallel computing; manage and coordinate the communication and data transmission between nodes to achieve efficient distributed computing, and is also responsible for task scheduling and management;

[0048] The task flow management module can assign tasks to appropriate nodes based on factors such as task priority, dependencies, and resource requirements, and monitor the execution and progress of tasks. It manages the state of the cluster through resource monitoring and task monitoring, and calls the remaining resources in the cluster for recalculation when a task fails. The goal is to improve the execution efficiency of tasks and the throughput of the overall system to ensure that tasks can be completed on time.

[0049] The task flow management module supports two scheduling rules: one is uniform scheduling, that is, scheduling to all nodes with the resource for calculation on average; the second is to allocate resources to nodes that meet the application according to the resource status of each node from large to small. According to the different needs of the task, the appropriate node can be dynamically selected and scheduled for calculation. For example, for tasks that require a lot of calculation, nodes with strong computing power can be given priority; for tasks that require a lot of storage, nodes with large storage capacity can be given priority.

[0050] The heterogeneous cluster resource management module is used to manage and call various hardware resources of different structures to form various nodes, identify various common physical or virtual hardware resources, support custom expansion management, support the configuration, monitoring and guarantee processing of hardware devices, and create the task management nodes, parameter service nodes and heterogeneous computing nodes defined by the task flow management module to form corresponding task management nodes, parameter service nodes and heterogeneous computing nodes; among them:

[0051] The task management node is used to manage heterogeneous resources and schedule heterogeneous tasks for the entire model distributed training task, which controls the entire training process;

[0052] The parameter service node is mainly used to receive local model information sent by heterogeneous computing nodes, aggregate model information, form updated global model information and send it to heterogeneous computing nodes;

[0053] The heterogeneous computing nodes receive the global training model information sent by the parameter service node, complete the local training task, update the local model information, and return the updated local model information to the parameter server.

[0054] The heterogeneous cluster resource management module uses Kubernetes to achieve unified management of cluster heterogeneous resources. It can run containerized applications on multiple physical or virtual machines. It also provides a disaster recovery mechanism for resources, automatically restarts failed containers, and migrates containers to other healthy nodes when a node fails.

[0055] The supported hardware devices include CPU, GPU, NPU, TPU, BPU, etc. For other intelligent computing hardware, when they have plug-ins that can be recognized by Kubernetes or Docker, they can be used in the form of dictionary declaration, such as {"other_resoureces": 3}. The declared resources can be called through the key of the declaration dictionary. Through this dictionary method, hardware resources can also be grouped for unification during resource scheduling. There are many ways to group:

[0056] The same hardware is allocated to the same resource group for management. For example, CPUs in the same group as GPUs are managed together, and CPUs in the same group as NPUs are managed together. Other third-party resources are managed together with CPUs in the same group, so as to achieve unified identification, monitoring, management, allocation, and recycling under heterogeneity.

[0057] The same resources can be divided into multiple resource groups, each of which performs specific tasks; each group has a computing scheduler for the corresponding resources, which is used for secondary scheduling of distributed tasks within the group. That is, the distributed tasks will be sent to each resource group through the heterogeneous task scheduler of the task management node. After receiving the tasks distributed by the heterogeneous task scheduler, the computing scheduler in each resource group will further subdivide the tasks and distribute them to all hardware resources in the resource group for computing.

[0058] Heterogeneous distributed clusters are built through the IP and port of the nodes. At the bottom layer, data and tasks are communicated and scheduled through gRPC. At the same time, all nodes in the cluster and the connected hardware resources are monitored. When a node or a piece of hardware fails, alarms and repairs can be implemented in time. gRPC is a remote procedure call (RPC) framework that allows a service to communicate between multiple servers, which can improve the communication efficiency between services and support the interoperability of multiple programming languages.

[0059] In this embodiment, the first-level task scheduling uses gRPC communication to send computing tasks to each resource group. The second-level task scheduling uses a high-speed communication channel and uses the ALLReduce method for calculation. If there is only one task or only one hardware resource in the second-level scheduling, the second-level communication is not required and only the first-level communication architecture is preserved.

[0060] This design method places larger communications on the secondary communication high-speed channel, reduces the frequency and scale of primary communications, improves overall task efficiency, and reduces communication pressure between nodes. At the same time, primary communication solves the problem of node hardware differences and realizes parallel computing of heterogeneous resources through hardware clusters.

[0061] The data distribution management module is used to manage and process the data information in the system, including at least pre-processing, segmentation, distribution management and round-robin of the data information, wherein:

[0062] Data preprocessing includes data size transformation, normalization, data enhancement, word segmentation and other operations to improve the accuracy, reliability and efficiency of data analysis and modeling. The specific operations depend on the data type and quality.

[0063] For example, for text data: cleaning operations such as removing special characters, punctuation marks, numbers, stop words, etc.; segmentation operations such as dividing the text into words, phrases or other units; and recombining continuous character sequences into word sequences according to certain specifications to facilitate computer processing.

[0064] For image data: resize the image by scaling or cropping it according to model requirements; normalize the image pixel values ​​from 0-255 to 0-1; generate new training samples for data enhancement based on image processing methods such as flipping, rotating, scaling, translating, cropping, color transformation, noise addition, blurring, and affine transformation.

[0065] The data distribution management module divides the entire amount of data for the training task and distributes it to each node. The data can be divided evenly or weighted according to the processing power of each node. One weighted method is to assume that there are N nodes in total, and the weighted distribution can be calculated based on the computing power of each node. The proportion Perform linear weighted segmentation on the entire data set.

[0066] The parallel strategy control module is used to complete heterogeneous distributed computing and model aggregation. It supports multiple strategy configurations and is flexibly adapted to various application scenarios.

[0067] The parallel strategy control module manages the control strategy of model convergence, adopts parallel types such as data parallelism and model parallelism, supports model convergence methods such as synchronous parameters and asynchronous parameters / gradients, and supports single-node single-card and single-node multi-card resource configuration modes.

[0068] After the data on all heterogeneous computing nodes completes an iterative training, the parallel strategy control module allows each heterogeneous computing node to transfer part of the data to its adjacent heterogeneous computing nodes, and optimizes the model training process through data polling, so that the model can be trained on a more balanced data distribution, improving the performance and generalization ability of the model. Specifically, after the data of each heterogeneous computing node completes an iterative training, for each heterogeneous computing node, a certain proportion of its data is randomly transferred to its adjacent heterogeneous computing nodes. data.

[0069] The distributed training method using the above heterogeneous parallel computing system includes the following steps:

[0070] Step 1: Create a distributed training task and submit it to a heterogeneous parallel computing system;

[0071] This step mainly includes the following sub-steps:

[0072] Step 11: Create a model development project. Users create a deep learning model development project as needed and record relevant project information, such as name, category, introduction, creator, and creation time.

[0073] Step 12: Define the model training task. After the user creates the deep learning model development project, define the related model training tasks, use the general computing framework to write the project code, and define the model, data set, optimizer, loss function, etc. required for stand-alone operation. The general computing frameworks include TensorFlow, PyTorch, MindSpore, and PaddlePaddle.

[0074] Step 13: Configure model training information. Users configure the computing resources, data splitting strategy and parallel training strategy required for distributed training for their defined deep learning model training tasks. Computing resources include CPU, GPU, NPU, TPU, etc. The data splitting strategy can be equal or weighted splitting. The parallel training strategy can be synchronous parameters and asynchronous parameters / gradients.

[0075] Step 14: Submit the model development task. After the user creates the deep learning model development project and configures the relevant model training task information, submit the model development task.

[0076] Step 2: The task flow management module parses the distributed training task, defines the corresponding task management node, parameter service node and heterogeneous computing node, and sends the resource demand information to the heterogeneous cluster resource management module to realize the conversion of the distributed training task;

[0077] Step 3: The heterogeneous cluster resource management module creates corresponding task management nodes, parameter service nodes, and heterogeneous computing nodes according to the analyzed task resource requirements;

[0078] Step 4: The task flow management module performs distributed training task scheduling: The heterogeneous task scheduler of the task management node divides the distributed training task according to the heterogeneous computing nodes, and sends the first-level task to each resource group. After receiving the first-level task distributed by the heterogeneous task scheduler, the computing scheduler in each resource group further subdivides the task into second-level tasks and distributes them to all hardware resources in the resource group. At the same time, the task flow management module sends the data segmentation strategy information to the data distribution management module, and sends the parallel training strategy information to the parallel strategy control module;

[0079] Step 5: The data distribution management module preprocesses, segments and distributes the corresponding data sets of the distributed training task; the preprocessing of the data set includes operations such as data size transformation, normalization, data enhancement, and word segmentation;

[0080] Training data segmentation and distribution includes uniformly segmenting the full amount of training task data according to the data segmentation strategy analyzed by the task process management module, and distributing it to various heterogeneous computing nodes;

[0081] Step 6: The parallel strategy control module performs corresponding distributed training; including: while the data distribution management module sends the model training data, all heterogeneous computing nodes start to synchronize the initial global model information of the parameter service node;

[0082] Based on the parallel training strategy information parsed by the task flow management module and the scheduling of distributed training tasks, the heterogeneous computing nodes perform local model training and interact with the parameter service nodes for model information to jointly complete the execution of the model distributed training tasks;

[0083] Step 7: The task process management module monitors the distributed training tasks in execution, including task status monitoring and training log monitoring. Users can view the status and log information of the model distributed training tasks as needed;

[0084] After the distributed model training task is completed, users can view the training results and model information;

[0085] Step 8: After the distributed training task is completed, the task process management module notifies the heterogeneous cluster resource management module to release the task management node, parameter service node and heterogeneous computing node corresponding to the distributed training task.

[0086] In a specific example of this application, the specific heterogeneous parallel distributed training process is as follows:

[0087] (1) The user creates and submits a model distributed training task:

[0088] ① Create a model development project: The user logs in to the model development page of the heterogeneous parallel computing system, selects the online development function, and creates an image detection project. The system records the relevant information of the project, including name, category, introduction, creator, and creation time.

[0089] ② Define model training tasks: Users initiate algorithm model development tasks based on specific business scenario requirements and data. Use the general computing framework Pytorch to write project code and define the model Resnet, data set CIFAR10, optimizer Adam, loss function CrossEntropyLoss, etc. required for stand-alone operation.

[0090] ③Configure model training information: The user configures the computing resources, data segmentation strategy, and parallel training strategy required for distributed training for the image detection task. For intelligent computing resources, select two GPU nodes with a total of 12 NVIDIA A100s, two NPU nodes with a total of 16 Ascend 910Bs, etc. For data segmentation strategy, select equal division, and for parallel training strategy, select synchronization parameter strategy.

[0091] ④Submit the model development task: After the user confirms the relevant model training task information of the image detection task, submit the model development task.

[0092] (2) Calling the heterogeneous parallel computing framework to perform model distributed training processing: The heterogeneous parallel computing framework is referenced to perform heterogeneous distributed training processing on the model training task.

[0093] ① The task flow management module converts the stand-alone code in step (1) into a distributed training task and provides parameters for the entire training process, such as setting the convergence step size to 20, performing model evaluation, and setting the log printing step size to 50. Define 1 task management node, 1 parameter service node, 2 GPU computing nodes, each with 6 NVIDIA A100s, and 2 NPU computing nodes, each with 8 Ascend 910Bs.

[0094] ② The heterogeneous cluster resource management module creates task management nodes, parameter service nodes, and heterogeneous work nodes, and connects the task management nodes, parameter server nodes, and heterogeneous work nodes through IP plus ports to build a logical heterogeneous computing cluster. One task management node, one parameter service node, two GPU computing nodes, and two NPU computing nodes are built into a resource pool dedicated to the image detection task to monitor and manage all hardware resources in the cluster.

[0095] ③The task flow management module performs distributed training task scheduling. The heterogeneous task scheduler of the task management node divides the distributed training task according to the heterogeneous computing nodes, divides it into 4 first-level tasks and sends them to the GPU computing node group and NPU computing node group respectively. The GPU computing scheduler of each GPU computing node subdivides its first-level task into 6 second-level tasks and distributes them to all hardware resources in the node. The NPU computing scheduler of each NPU computing node subdivides its first-level task into 8 second-level tasks and distributes them to all hardware resources in the node. At the same time, the task flow management module sends the data segmentation strategy information to the data distribution management module and sends the parallel training strategy information to the parallel strategy control module.

[0096] ④ The data distribution management module first loads data on the parameter service node of the logical heterogeneous computing cluster, evenly divides the data and distributes it to all heterogeneous computing nodes through HTTPS. At the same time as the data is sent, all heterogeneous computing nodes begin to synchronize the model initialized by the parameter service node. After the heterogeneous computing nodes complete synchronization and the task management node data distribution is completed, the business process tasks defined in the task process management module are called in sequence.

[0097] ⑤ The parallel strategy control module configures the parallel strategy of synchronization parameters. Each heterogeneous computing node has a certain waiting time to ensure that after all heterogeneous computing nodes have completed the tasks of the same batch, they can calculate the next batch and finally complete the training task of the entire model. After the data of all heterogeneous computing nodes complete an iterative training, for each heterogeneous computing node, 5% of the data is randomly transferred to its adjacent heterogeneous computing nodes.

[0098] (3) Model distributed training task execution and monitoring: After calling the heterogeneous parallel computing framework for model distributed training, the system executes heterogeneous distributed computing for the image detection task.

[0099] The heterogeneous computing node receives the corresponding task and transmits the task result to the parameter service node for aggregation and update according to the defined business process after the task is completed. When the training result meets the threshold or the number of iterations reaches the upper limit, the task is terminated. The task process management module notifies the heterogeneous cluster resource management module to release all resources in the dedicated cluster for the image detection task. Users can view the training results and model information.

[0100] The beneficial effects of adopting the heterogeneous parallel computing system of this embodiment are: the heterogeneous parallel computing system supports high scalability, supports multiple general computing frameworks, and supports multiple types of intelligent computing hardware devices;

[0101] Ability to split a deep learning heterogeneous task and distribute it to intelligent computing hardware of different resource types, execute tasks concurrently, and improve computing performance and efficiency;

[0102] Heterogeneous parallel computing systems can make full use of intelligent computing hardware of different resource types and adapt to more AI model development needs. They can use homogeneous AI computing cards to execute distributed training of models as needed, or they can use heterogeneous AI computing cards to execute distributed training of models.

[0103] It should be noted that the above detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the art to which the present application belongs.

[0104] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0105] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0106] In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0107] For ease of description, spatially relative terms, such as "above", "above", "on the upper surface of", "above", etc., may be used herein to describe the spatial positional relationship between a device or feature and other devices or features as shown in the figure. It should be understood that spatially relative terms are intended to include different orientations of the device in use or operation in addition to the orientation described in the figure. For example, if the device in the accompanying drawings is inverted, the device described as "above other devices or structures" or "above other devices or structures" will be positioned as "below other devices or structures" or "below other devices or structures". Thus, the exemplary term "above" may include both "above" and "below". The device may also be positioned in other different ways, such as rotated 90 degrees or in other orientations, and the spatially relative descriptions used herein are interpreted accordingly.

[0108] In the above detailed description, reference is made to the accompanying drawings, which form a part of this document. In the accompanying drawings, similar symbols typically identify similar components unless the context indicates otherwise. The illustrated embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be used, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A heterogeneous parallel computing system, characterized in that: It includes task process management module, heterogeneous cluster resource management module, data distribution management module and parallel strategy control module, among which: The task flow management module is used to parse the model distributed training task, define the corresponding nodes, including the task management node, the parameter service node and the heterogeneous computing node, decompose the training task into multiple subtasks, and assign these subtasks to different nodes for parallel computing; The heterogeneous cluster resource management module is used to manage and call hardware resources of different structures to form the nodes, monitor and ensure the hardware resources; the hardware resources of various structures include CPU, GPU, NPU, TPU and BPU, and the hardware resources are divided into multiple resource groups for management. The same hardware resources are allocated in the same resource group. There are several resource groups for the same hardware resources, and each resource group forms a node; the nodes build heterogeneous distributed clusters through IP and ports, the first-level task scheduling uses gRPC communication to send computing tasks to each resource group, and the second-level task scheduling uses high-speed communication channels and uses ALLReduce to perform calculations; and monitor all nodes and connected hardware resources in the cluster, and when a node or a hardware resource fails, it can alarm, repair or migrate; The data distribution management module is used to manage and process the data information in the system, at least including preprocessing, segmentation, distribution management and polling of the data information; the data distribution management module preprocesses the data information including: size transformation, normalization, data enhancement and word segmentation of the data information; The splitting process includes average splitting or weighted splitting of data information, where weighted splitting is based on the proportion of each node's total computing power to all nodes; Distribution management is to split all the data information generated by the training task and send it to each node; The parallel strategy control module is used to complete heterogeneous distributed computing and model convergence; after the data information on all heterogeneous computing nodes completes one iterative training, the parallel strategy control module allows each heterogeneous computing node to randomly transfer a certain proportion of data to an adjacent heterogeneous computing node.

2. The heterogeneous parallel computing system according to claim 1, characterized in that: The task flow management module allocates the training task to the node by uniform scheduling or resource status scheduling according to the priority, dependency and resource requirement of the training task; wherein: The uniform scheduling is to evenly schedule all the nodes that own the resource for calculation; The resource status scheduling is to allocate the resources of each node in descending order to the nodes that meet the training task.

3. A distributed training method using the heterogeneous parallel computing system of claim 1 or 2, characterized in that: The steps include: Step 1: Create a distributed training task and submit it to a heterogeneous parallel computing system; Step 2: The task flow management module parses the distributed training task, defines the corresponding task management nodes, parameter service nodes and heterogeneous computing nodes, and sends the resource demand information to the heterogeneous cluster resource management module; Step 3: The heterogeneous cluster resource management module creates corresponding task management nodes, parameter service nodes, and heterogeneous computing nodes according to the analyzed task resource requirements; Step 4: The task flow management module performs distributed training task scheduling. The task management node divides the distributed training tasks according to the heterogeneous computing nodes and sends them to each resource group. At the same time, the data segmentation strategy information is sent to the data distribution management module, and the parallel training strategy information is sent to the parallel strategy control module. Step 5: The data distribution management module preprocesses, splits and distributes the corresponding data sets of the distributed training task; Step 6: The parallel strategy control module performs corresponding distributed training; Step 7: The task process management module monitors the distributed training tasks in execution, including task status monitoring and training log monitoring; Step 8: After the distributed training task is completed, the task process management module notifies the heterogeneous cluster resource management module to release the task management node, parameter service node and heterogeneous computing node corresponding to the distributed training task.

4. The distributed training method according to claim 3, characterized in that: In step 1, specifically include: Step 11: Create a model development project as needed and record relevant project information; Step 12: Define the relevant model training tasks, write project code using a general computing framework, and define the model, data set, optimizer, and loss function required for stand-alone operation; Step 13: Configure the computing resources, data segmentation strategy, and parallel training strategy required for distributed training for the model training task; Step 14: Submit the model development task to the heterogeneous parallel computing system.

5. The distributed training method according to claim 3, characterized in that: In step 4, the task management node divides the distributed training tasks according to the heterogeneous computing nodes, and sends the first-level tasks to each resource group. Each resource group divides the first-level tasks into several second-level tasks, and distributes the second-level tasks to all hardware resources in the resource group.

6. The distributed training method according to claim 3, characterized in that: In step 6, while the data distribution management module sends the model training data, all heterogeneous computing nodes begin to synchronize the initial global model information of the parameter service node; Based on the parallel training strategy information parsed by the task flow management module and the scheduling of distributed training tasks, heterogeneous computing nodes perform local model training and interact with parameter service nodes for model information to jointly complete the execution of model distributed training tasks.

Images