Node role allocation method and device in distributed training and electronic equipment

Abstract

The invention provides a node role allocation method and device in distributed training and electronic equipment, relates to the technical field of artificial intelligence, and is used for realizing collaborative optimization of a cluster resource utilization rate and distributed training efficiency and remarkably improving system robustness, fault tolerance and automation degree. The method comprises the following steps: constructing a computing power network resource map based on computing power resource information of computing power equipment in a cluster; wherein nodes of the computing power network resource map represent computing power equipment, and attributes of the nodes comprise resource states; generating a node role allocation scheme for the to-be-executed distributed training task based on the computing power network resource map; distributing a service node role or a working node role for each computing power device in the cluster based on the node role distribution scheme, and starting a distributed training task; monitoring training performance data in a distributed training process; and optimizing the node role allocation scheme based on the training performance data, and performing node role reallocation based on the optimized node role allocation scheme.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus and electronic devices for assigning node roles in distributed training. Background Technology

[0002] With the rapid development of artificial intelligence technology, the scale of deep learning model parameters has grown dramatically, and its applications in fields such as computer vision, natural language processing, and scientific computing have become increasingly widespread and in-depth. This trend has placed unprecedented demands on computing power, making traditional single-machine training modes unable to meet the needs of large-scale model training. As a result, distributed training has become the mainstream model training paradigm to adapt to the computational challenges of massive amounts of data and complex models.

[0003] Among various distributed training architectures, the parameter server architecture is favored in heterogeneous computing environments due to its simple design and strong fault tolerance. This architecture typically includes worker nodes responsible for local computation and server nodes responsible for global model aggregation. However, in actual deployments, there are often significant differences in the hardware performance (such as GPU computing power and memory), network bandwidth, and latency of computing clusters, posing numerous challenges to node role allocation: resource heterogeneity leads to an imbalance in computing / storage resource allocation; high latency during dynamic scaling up and down; weak fault tolerance mechanisms easily lead to single points of failure; static load balancing strategies are difficult to cope with real-time fluctuations; the disconnect between network topology and role allocation results in unoptimized communication latency; and low automation relies on manual configuration. These problems collectively restrict the efficiency and resource utilization of distributed training. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for assigning node roles in distributed training, which can achieve synergistic optimization of cluster resource utilization and distributed training efficiency, and significantly improve system robustness, fault tolerance, and automation.

[0005] Firstly, this application provides a method for assigning node roles in distributed training, including: Based on the computing power resource information of computing devices in the cluster, a computing power network resource graph is constructed; wherein, the nodes of the computing power network resource graph represent the computing power devices, and the attributes of the nodes include resource status; the edges of the computing power network resource graph represent the network connections between the computing power devices, and the attributes of the edges include communication performance. Based on the computing power network resource map, a node role allocation scheme is generated for the distributed training task to be executed; Based on the node role allocation scheme, service node roles or worker node roles are assigned to each computing device in the cluster, and the distributed training task is started. Monitor training performance data during distributed training; The node role allocation scheme is optimized based on the training performance data, and node roles are reassigned based on the optimized node role allocation scheme.

[0006] The technical solution provided in this application brings at least the following beneficial effects: By constructing a computing power network resource map that integrates resource status and communication performance, precise perception of heterogeneous cluster resources is achieved, laying a data foundation for scientific decision-making. Based on this, a node role allocation scheme is automatically generated, fundamentally solving the resource imbalance and communication latency problems caused by resource heterogeneity and network topology disconnect, achieving synergistic optimization of resource utilization and training efficiency. Then, through a closed-loop mechanism that monitors performance data and dynamically triggers role reassignment, the system can respond in real time to node failures, load fluctuations, and scaling requirements, upgrading static strategies to adaptive optimization, thereby significantly enhancing the system's robustness and fault tolerance. Finally, the above-mentioned fully automated process from perception and decision-making to optimization completely overcomes the dependence on manual configuration, achieving a high degree of automation in distributed training resource allocation and operation. Therefore, this application can achieve synergistic optimization of cluster resource utilization and distributed training efficiency, and significantly improve the system's robustness, fault tolerance, and automation level.

[0007] One possible implementation is that the resource status includes the status of heterogeneous computing resources, storage resources, and network resources, and the communication performance includes network communication quality and network communication topology.

[0008] One possible implementation involves generating a node role allocation scheme for the distributed training task to be executed based on the computing power network resource graph, including: Based on the resource status of each node in the computing power network resource graph, a set of candidate working nodes that meet the computing resource requirements of the distributed training task to be executed is selected from the computing power network resource graph. Based on network centrality, a set of candidate service nodes is selected from nodes that have network connections with the set of candidate worker nodes; With the goal of minimizing cross-role communication overhead and balancing the load among nodes, a multi-objective optimization algorithm is used to assign corresponding working nodes to each candidate service node in the candidate service node set, thereby generating the node role allocation scheme.

[0009] One possible implementation, wherein selecting a set of candidate service nodes from nodes that have network connections with the set of candidate worker nodes based on network centrality, includes: From the computing power network resource map, target nodes that have network connections with the candidate set of working nodes are selected as the service node candidate range; Based on the communication performance of the edges between the target nodes within the candidate range of the service node, the network centrality index of each target node is calculated. Based on the network centrality index of each target node, the target nodes are sorted in descending order, and the target nodes with the highest number of targets are selected to form the candidate service node set.

[0010] One possible implementation, wherein the goal is to minimize cross-role communication overhead and balance the load among nodes, a multi-objective optimization algorithm is used to assign corresponding worker nodes to each candidate service node in the candidate service node set, including: Generate random chromosomes; wherein each chromosome is an integer array, and each element in the integer array represents the number of the candidate service node assigned to the candidate worker node in the candidate worker node set; Construct a first objective function and a second objective function; the first objective function is used to minimize the maximum communication overhead between each candidate worker node and the assigned candidate service node; the second objective function is used to minimize the imbalance between the total computing power of all candidate worker nodes to which each candidate service node is assigned. Set constraints; the constraints are that the total number of the selected candidate working nodes in the chromosome is equal to the required number of working nodes. A non-dominated sorting genetic algorithm is used to solve the multi-objective optimization problem.

[0011] One possible implementation involves optimizing the node role allocation scheme based on the training performance data, including: Compare each performance metric in the training performance data with the corresponding preset threshold; Determine the target performance index that deviates from the preset threshold; Based on the degree of deviation between the target performance index and the preset threshold, the node role allocation scheme is optimized; the optimized node role allocation scheme includes the reselection of service nodes or worker nodes, or the redistribution of the affiliation relationship between worker nodes and service nodes.

[0012] One possible implementation of the method further includes: Periodically re-examine the computing resource information of the computing devices in the cluster; The computing network resource map is updated using the computing resource information of the computing devices in the cluster that have been re-explored.

[0013] Secondly, this application provides a node role allocation device in distributed training, comprising: A construction module is used to construct a computing power network resource graph based on the computing power resource information of computing power devices in the cluster; wherein, the nodes of the computing power network resource graph represent the computing power devices, and the attributes of the nodes include resource status; the edges of the computing power network resource graph represent the network connections between the computing power devices, and the attributes of the edges include communication performance. The generation module is used to generate a node role allocation scheme for the distributed training task to be executed based on the computing power network resource map. The allocation module is used to allocate service node roles or worker node roles to each computing device in the cluster based on the node role allocation scheme, and to start the distributed training task. The monitoring module is used to monitor training performance data during distributed training. The optimization module is used to optimize the node role allocation scheme based on the training performance data; The reassignment module is used to reassign node roles based on the optimized node role allocation scheme.

[0014] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory storing processor-executable instructions; when the processor is configured to execute the instructions, causing the electronic device to implement the method of the first aspect described above.

[0015] Fourthly, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to implement the method described in the first aspect.

[0016] Fifthly, this application provides a computer program product comprising a computer program; when the computer program is run in an electronic device, the electronic device performs the method described in the first aspect.

[0017] The beneficial effects of the second to fifth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a node role allocation method in distributed training provided in this application embodiment; Figure 2A schematic diagram illustrating the composition of a node role allocation device in distributed training provided in this application embodiment; Figure 3 This is a schematic diagram of the composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The following section will describe in detail a node role allocation method in distributed training provided in this application, with reference to the accompanying drawings.

[0020] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first" and "second," etc., used in the specification and drawings of this application are used to distinguish different objects or to distinguish different treatments of the same object, rather than to describe a specific order of objects.

[0022] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus.

[0023] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0024] To facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with essentially the same function and effect. Those skilled in the art can understand that the terms "first" and "second" are not intended to limit the quantity or execution order.

[0025] In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0026] See Figure 1 This is a flowchart illustrating a node role allocation method in distributed training provided in an embodiment of this application. Figure 1As shown, the specific steps include S101 to S105.

[0027] S101. Construct a computing network resource map based on the computing resource information of computing devices in the cluster.

[0028] In the computing power network resource graph, nodes represent computing power devices, and the attributes of nodes include resource status; edges represent network connections between computing power devices, and the attributes of edges include communication performance.

[0029] Specifically, by collecting computing resource information such as hardware configuration (e.g., computing power type, memory capacity) and real-time status (e.g., current load) of each computing device (e.g., GPU, CPU, NPU, etc.) in the cluster, each computing device is abstracted as a node, and the network connection between nodes is abstracted as an edge. This constructs a computing network resource map that integrates multi-dimensional attributes such as computing power, storage, and network topology. This enables accurate perception of heterogeneous cluster resources, lays a data foundation for scientific decision-making, and fundamentally solves the management problems caused by resource heterogeneity.

[0030] In some embodiments, before task scheduling, a benchmarking tool is run to explore the computing resource information of computing devices in the cluster from multiple dimensions.

[0031] For example, at the computing resource layer, Node Exporter, NVIDIA DCGM, and various AI chip-specific monitoring tools are integrated, and Prometheus is used to collect basic performance indicators of heterogeneous computing power, including CPUs, GPUs, and accelerator cards. At the network layer, SolarWinds, NetDoctor, and iperf3 are comprehensively utilized to achieve automatic discovery of network topology and continuous monitoring of key performance indicators such as bandwidth and latency. At the storage layer, Ceph Dashboard and Prometheus are used to collect the health status and performance indicators of the storage cluster. All data sources from all layers are uniformly aggregated and stored in Prometheus, and an integrated resource monitoring dashboard is built based on Grafana. This system also provides rich API interfaces to provide data support for subsequent node role allocation and task scheduling decisions based on multi-dimensional computing resource information.

[0032] After exploring the multi-dimensional computing power resource information of all computing devices in the cluster, a computing power network resource map is constructed.

[0033] The nodes in the computing power network resource graph represent computing devices, and the attributes of a node include its resource status. Optionally, the resource status includes the status of heterogeneous computing resources, storage resources, and network resources. Each node (potential Server / Worker) is a vertex, and each vertex... It possesses key attributes.

[0034] For example, the key attributes of a node can be shown in Table 1.

[0035] Table 1 Key Attributes of Nodes

[0036] The edges of the computing power network resource graph represent the network connections between computing power devices, and the attributes of the edges include communication performance.

[0037] Optionally, communication performance includes network communication quality and network communication topology. Each pair of nodes in the computing power network resource map... and There is an edge between them. The edge possesses key attributes.

[0038] For example, the key properties of an edge can be shown in Table 2.

[0039] Table 2 Key properties of edges

[0040] S102. Based on the computing power network resource graph, generate a node role allocation scheme for the distributed training task to be executed.

[0041] Specifically, based on the computing power network resource map, a node role allocation scheme is generated for the distributed training tasks to be executed, which solves the problems of resource imbalance and communication delay caused by resource heterogeneity and network topology disconnection from the source, and realizes the synergistic optimization of resource utilization and training efficiency.

[0042] In some embodiments, step S102 includes the following sub-steps S1021-S1023.

[0043] S1021. Based on the resource status of each node in the computing power network resource graph, select a set of candidate working nodes from the computing power network resource graph that meet the computing resource requirements of the distributed training task to be executed.

[0044] For example, firstly, based on the number of worker nodes specified by the user... Based on the worker node model (e.g., A100 or H100), and the resource status of nodes in the computing power network resource map (e.g., node availability, basic attributes, and performance attributes), candidate nodes that meet the computing resource requirements of the distributed training task to be executed in terms of computing power and memory capacity are selected. Subsequently, the candidate nodes are ranked according to network topology and performance indicators, prioritizing node groups with similar physical or regional locations, and prioritizing nodes supporting high-speed RDMA network technologies such as RoCE, ultimately resulting in a list containing... A set of candidate working nodes (in ), used for subsequent selection.

[0045] S1022. Based on network centrality, select a set of candidate service nodes from the nodes that have network connections with the set of candidate worker nodes.

[0046] Specifically, nodes with network connections to the candidate set of worker nodes are selected based on the computing power network resource map, and the network centrality index of these nodes in the network topology is calculated. Based on the network centrality index of these nodes in the network topology, multiple communication "hub" nodes are identified as a candidate set of service nodes, thereby minimizing the comprehensive communication path overhead between all worker nodes and service nodes in the topology structure, laying the foundation for fundamentally reducing the latency bottleneck of parameter synchronization.

[0047] Optionally, step S1022 includes: selecting target nodes that have network connections with the candidate set of working nodes from the computing power network resource graph, as the service node candidate range; calculating the network centrality index of each target node based on the communication performance of the edges between each target node in the service node candidate range; sorting each target node in descending order based on the network centrality index of each target node, and selecting the target nodes with the highest number of target nodes to form a candidate service node set.

[0048] For example, firstly, the number of candidate service nodes is determined based on empirical rules. For example: The constraint is the number of candidate working nodes. A certain percentage (e.g., not exceeding 20%), while greater than , To determine the number of working nodes specified by the user The number of servers is constrained, for example, by selecting a certain proportion. Then, a set of candidate worker nodes is selected from the computing power network resource graph. Directly connected nodes. If there are not enough directly connected candidate nodes, the pool is expanded to include indirectly connected nodes, forming the candidate range for service nodes. Target nodes (excluding nodes already selected as workers). Then, based on the communication performance attributes of the network connections between each target node within the candidate service node range (such as a comprehensive communication capability score). Calculate path weights, and use these path weights to calculate the network centrality metric (e.g., betweenness centrality or tight centrality) of each target node. Finally, rank the target nodes according to their network centrality metrics and then... Select the highest-ranking target node. A set of candidate service nodes is formed by selecting target nodes as key "hub" nodes. By defining path weights based on communication performance attributes, network centrality calculation is transformed from simple "physical topology analysis" to "network performance analysis." This allows for the intelligent selection of service nodes that can truly minimize global communication latency, directly solving the problem of "disconnect between network topology and role allocation" and achieving deep optimization of communication latency.

[0049] S1023. With the goal of minimizing cross-role communication overhead and balancing the load among nodes, a multi-objective optimization algorithm is used to assign corresponding working nodes to each candidate service node in the candidate service node set, thereby generating a node role allocation scheme.

[0050] Specifically, the allocation problem of worker nodes and service nodes is abstracted into a multi-objective optimization problem. The two competing objectives are minimizing cross-role communication overhead (i.e., reducing the maximum communication latency between worker nodes and their assigned service nodes) and balancing the load among nodes (i.e., reducing the difference in total computing power among worker nodes under each service node). A multi-objective optimization algorithm (such as NSGA-II) is used to solve this problem. By searching for the optimal allocation combination, worker nodes are rationally allocated among multiple candidate service nodes. Finally, a node role allocation scheme that achieves the best balance between communication efficiency and computational load is output, thereby systematically improving the overall efficiency of distributed training.

[0051] Optionally, step S1023 includes: generating random chromosomes; wherein each chromosome is an integer array, and each element in the integer array represents the number of the candidate service node assigned to the candidate worker node in the candidate worker node set; constructing a first objective function and a second objective function; the first objective function is used to minimize the maximum communication overhead between each candidate worker node and the assigned candidate service node; the second objective function is used to minimize the imbalance between the total computing power of all candidate worker nodes assigned to each candidate service node; setting constraints; the constraint is that the total number of selected candidate worker nodes in the chromosome is equal to the required number of worker nodes; and using a non-dominated sorting genetic algorithm to solve the multi-objective optimization problem.

[0052] For example, multi-objective optimization is performed through the following steps: 1) Encoding: Generate random chromosomes, each chromosome being of length [missing information]. An integer array whose elements take values ​​in the range of 1 / 2. 0 indicates that the corresponding candidate worker node was not selected, and non-zero values ​​are not allowed. This indicates the number of the candidate service node assigned to the candidate worker node in the candidate worker node set; for example, 1 indicates that the corresponding candidate worker node is assigned to the candidate service node with the number 1; 2) Objective function: The first objective function aims to minimize the maximum communication overhead between each candidate worker node and the assigned candidate service node. , This indicates the available bandwidth of the network link between the candidate worker node and the assigned candidate service node. This represents the round-trip latency of the network link between the candidate worker node and the assigned candidate service node. and Indicates the weighting coefficient; and The second objective function is used to adjust the relative importance of bandwidth and latency in the overall communication overhead. Its specific value can be adjusted according to the actual network environment and the sensitivity of the training task to bandwidth or latency. For example, in tasks with a large number of model parameters, a higher weight may be given to the bandwidth term (i.e., increasing α); while in synchronous training with frequent communication, more attention may be paid to latency (i.e., increasing β). The second objective function is used to minimize the imbalance (e.g., variance or standard deviation) between the total computing power of all candidate worker nodes allocated to each candidate service node, in order to achieve load balancing. 3) Constraints: The appropriate values ​​must be selected. There are 10 working nodes, which ensures that the number of non-zero genes in the chromosome is 1. ;4) Use a non-dominated sorting genetic algorithm (such as NSGA-II) to solve multi-objective optimization problems, thereby determining the best solution.

[0053] In this embodiment, firstly, by constructing a first objective function aimed at minimizing maximum communication overhead and quantifying communication costs based on link bandwidth and latency, the data transmission efficiency between worker nodes and service nodes is directly optimized along the communication path, effectively reducing synchronization waiting time caused by network bottlenecks and thus directly improving the iteration speed of distributed training. Secondly, by constructing a second objective function aimed at minimizing the difference in total computing power (imbalance) among worker nodes under each service node, the computational load is ensured to be evenly distributed among different service nodes, avoiding resource waste caused by some nodes being overloaded while others are idle, thereby significantly improving the overall computing resource utilization of the cluster. Finally, a non-dominated sorting genetic algorithm is used for multi-objective optimization, achieving the best trade-off between competing communication efficiency and load balancing objectives, generating a globally optimal node role allocation scheme, thus systematically realizing the synergistic optimization of distributed training efficiency and resource utilization.

[0054] S103. Based on the node role allocation scheme, assign service node roles or worker node roles to each computing device in the cluster, and start the distributed training task.

[0055] Specifically, the generated node role allocation scheme is parsed and transformed into specific system instructions. The scheduling system (such as Kubernetes) dynamically configures the roles of the corresponding computing devices in the cluster according to the scheme: specific nodes are marked and initialized as service nodes (responsible for parameter aggregation), and the remaining nodes are marked and initialized as worker nodes (responsible for local gradient calculation). Based on these roles, corresponding training task parameters and environment dependencies are configured. Finally, all nodes are coordinated to load the model and data, officially starting the distributed training task, enabling the entire system to begin collaborative work according to the preset optimization scheme. This process achieves a closed loop from intelligent decision-making to automated execution, completely eliminating the inefficient process of manually configuring nodes.

[0056] S104. Monitor training performance data during distributed training.

[0057] S105. Optimize the node role allocation scheme based on training performance data, and redistribute node roles based on the optimized node role allocation scheme.

[0058] Specifically, the performance metrics in the training performance data are compared with their corresponding preset thresholds; target performance metrics that deviate from the preset thresholds are determined; based on the degree of deviation between the target performance metrics and the preset thresholds, the node role allocation scheme is optimized; the optimized node role allocation scheme includes the reselection of service nodes or worker nodes, or the redistribution of the affiliation between worker nodes and service nodes. For example, if monitoring finds that the actual communication time of a certain worker-server link continuously exceeds a preset threshold, the worker can be dynamically reassigned to another server node that is more idle or has a better network condition.

[0059] In some embodiments, the method further includes: periodically (e.g., every 15 minutes) re-probing the computing resource information of computing devices in the cluster; and updating the computing network resource graph using the re-probing computing resource information of computing devices in the cluster. By periodically updating the computing network resource graph, it can be ensured that role allocation decisions are always based on the real-time status of the cluster, thereby ensuring the continuous, efficient and stable operation of distributed training tasks.

[0060] The technical solutions provided in the above embodiments bring at least the following beneficial effects: The node role allocation method in distributed training provided in this application constructs a computing power network resource map that integrates resource status and communication performance, achieving accurate perception of heterogeneous cluster resources and laying a data foundation for scientific decision-making. Based on this, a node role allocation scheme is automatically generated, fundamentally solving the resource imbalance and communication latency problems caused by resource heterogeneity and network topology disconnect, achieving synergistic optimization of resource utilization and training efficiency. Then, by monitoring performance data and dynamically triggering a closed-loop mechanism for role reassignment, the system can respond in real time to node failures, load fluctuations, and scaling requirements, upgrading static strategies to adaptive optimization, thereby significantly enhancing the system's robustness and fault tolerance. Finally, the above-mentioned fully automated process from perception and decision-making to optimization completely overcomes the dependence on manual configuration, achieving a high degree of automation in distributed training resource allocation and operation. Therefore, this application can achieve synergistic optimization of cluster resource utilization and distributed training efficiency, and significantly improve the system's robustness, fault tolerance, and automation level.

[0061] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0062] This application embodiment can divide the node role allocation device in distributed training into functional modules according to the above method example. For example, each function can be divided into a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0063] In some embodiments, this application also provides a node role allocation device in distributed training. This node role allocation device in distributed training may include one or more functional modules for implementing the node role allocation method in distributed training described in the above method embodiments.

[0064] For example, Figure 2 This is a schematic diagram illustrating the composition of a node role allocation device in distributed training, as provided in an embodiment of this application. Figure 2 As shown, the node role allocation device 200 in the distributed training includes: Module 201 is used to construct a computing power network resource graph based on the computing power resource information of computing power devices in the cluster. Nodes in the computing power network resource graph represent computing power devices, and the attributes of nodes include resource status. Edges in the computing power network resource graph represent network connections between computing power devices, and the attributes of edges include communication performance. The generation module 202 is used to generate a node role allocation scheme for the distributed training task to be executed based on the computing power network resource map. The allocation module 203 is used to allocate service node roles or worker node roles to each computing device in the cluster based on the node role allocation scheme, and to start distributed training tasks. Monitoring module 204 is used to monitor training performance data during distributed training. Optimization module 205 is used to optimize the node role allocation scheme based on training performance data; The reassignment module 206 is used to reassign node roles based on the optimized node role allocation scheme.

[0065] In some embodiments, resource status includes the status of heterogeneous computing resources, storage resources, and network resources, and communication performance includes network communication quality and network communication topology.

[0066] In some embodiments, the generation module 202 includes: The first screening unit is used to select a set of candidate working nodes that meet the computing resource requirements of the distributed training task to be executed from the computing power network resource graph based on the resource status of each node in the computing power network resource graph. The second filtering unit is used to filter out a set of candidate service nodes from nodes that have network connections with the set of candidate worker nodes based on network centrality. The allocation unit is used to generate a node role allocation scheme by employing a multi-objective optimization algorithm to allocate corresponding working nodes to each candidate service node in the candidate service node set, with the goal of minimizing cross-role communication overhead and balancing the load among nodes.

[0067] In some embodiments, the second filtering unit is specifically used for: From the computing power network resource map, target nodes that have network connections with the candidate set of working nodes are selected as the range of candidate service nodes; Based on the communication performance of the edges between each target node within the candidate range of the service node, the network centrality index of each target node is calculated. Based on the network centrality index of each target node, the target nodes are sorted in descending order, and the target nodes with the highest number of target nodes are selected to form a candidate service node set.

[0068] In some embodiments, the allocation unit is specifically used for: Generate random chromosomes; where each chromosome is an integer array, and each element in the integer array represents the number of the candidate service node assigned to the candidate worker node in the candidate worker node set; Construct a first objective function and a second objective function; the first objective function is used to minimize the maximum communication overhead between each candidate worker node and the assigned candidate service node; the second objective function is used to minimize the imbalance between the total computing power of all candidate worker nodes assigned to each candidate service node. Set constraints; the constraint is that the total number of selected candidate working nodes in the chromosome is equal to the required number of working nodes. A non-dominated sorting genetic algorithm is used to solve the multi-objective optimization problem.

[0069] In some embodiments, the optimization module 205 is specifically used for: Compare each performance metric in the training performance data with the corresponding preset thresholds; Determine the target performance index that deviates from the preset threshold; Based on the deviation between the target performance index and the preset threshold, the node role allocation scheme is optimized; the optimized node role allocation scheme includes the reselection of service nodes or worker nodes, or the redistribution of the affiliation relationship between worker nodes and service nodes.

[0070] In some embodiments, the device further includes an update module, specifically configured to: Periodically re-examine the computing resource information of computing devices in the cluster; The computing power network resource map is updated by utilizing the computing power resource information of the computing power devices in the re-explored cluster.

[0071] When implementing the functions of the integrated modules described above in hardware, this embodiment of the invention provides a possible schematic diagram of the electronic device involved in the above embodiments. For example... Figure 3 As shown, the electronic device 300 includes: a processor 302, a communication interface 303, and a bus 304. Optionally, the electronic device 300 may also include a memory 301.

[0072] Processor 302 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 302 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 302 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0073] Communication interface 303 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0074] The memory 301 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0075] As one possible implementation, the memory 301 can exist independently of the processor 302. The memory 301 can be connected to the processor 302 via a bus 304 and is used to store instructions or program code. When the processor 302 calls and executes the instructions or program code stored in the memory 301, it can implement the node role allocation method in distributed training provided in this embodiment of the invention.

[0076] In another possible implementation, the memory 301 can also be integrated with the processor 302.

[0077] Bus 304 can be an extended industry standard architecture (EISA) bus, etc. Bus 304 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0078] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the service calling device can be divided into different functional modules to complete all or part of the functions described above.

[0079] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned service invocation device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned service invocation device. Further, the aforementioned computer-readable storage medium can include both internal storage units of the aforementioned service invocation device and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned service invocation device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0080] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to execute any of the node role allocation methods in distributed training provided in the above embodiments.

[0081] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for assigning node roles in distributed training, characterized in that, include: Based on the computing power resource information of computing devices in the cluster, a computing power network resource graph is constructed; wherein, the nodes of the computing power network resource graph represent the computing power devices, and the attributes of the nodes include resource status; the edges of the computing power network resource graph represent the network connections between the computing power devices, and the attributes of the edges include communication performance. Based on the computing power network resource map, a node role allocation scheme is generated for the distributed training task to be executed; Based on the node role allocation scheme, service node roles or worker node roles are assigned to each computing device in the cluster, and the distributed training task is started. Monitor training performance data during distributed training; The node role allocation scheme is optimized based on the training performance data, and node roles are reassigned based on the optimized node role allocation scheme.

2. The node role allocation method in distributed training according to claim 1, characterized in that, The resource status includes the status of heterogeneous computing resources, storage resources, and network resources, and the communication performance includes network communication quality and network communication topology.

3. The node role allocation method in distributed training according to claim 1, characterized in that, The step of generating a node role allocation scheme for the distributed training task to be executed based on the computing power network resource graph includes: Based on the resource status of each node in the computing power network resource graph, a set of candidate working nodes that meet the computing resource requirements of the distributed training task to be executed is selected from the computing power network resource graph. Based on network centrality, a set of candidate service nodes is selected from nodes that have network connections with the set of candidate worker nodes; With the goal of minimizing cross-role communication overhead and balancing the load among nodes, a multi-objective optimization algorithm is used to assign corresponding working nodes to each candidate service node in the candidate service node set, thereby generating the node role allocation scheme.

4. The node role allocation method in distributed training according to claim 3, characterized in that, The step of selecting a set of candidate service nodes from nodes that have network connections with the set of candidate worker nodes based on network centrality includes: From the computing power network resource map, target nodes that have network connections with the candidate set of working nodes are selected as the service node candidate range; Based on the communication performance of the edges between the target nodes within the candidate range of the service node, the network centrality index of each target node is calculated. Based on the network centrality index of each target node, the target nodes are sorted in descending order, and the target nodes with the highest number of targets are selected to form the candidate service node set.

5. The node role allocation method in distributed training according to claim 3, characterized in that, The process of assigning corresponding working nodes to each candidate service node in the candidate service node set, with the objectives of minimizing cross-role communication overhead and balancing the load among nodes, employs a multi-objective optimization algorithm, including: Generate random chromosomes; wherein each chromosome is an integer array, and each element in the integer array represents the number of the candidate service node assigned to the candidate worker node in the candidate worker node set; Construct a first objective function and a second objective function; the first objective function is used to minimize the maximum communication overhead between each candidate worker node and the assigned candidate service node; the second objective function is used to minimize the imbalance between the total computing power of all candidate worker nodes to which each candidate service node is assigned. Set constraints; the constraints are that the total number of the selected candidate working nodes in the chromosome is equal to the required number of working nodes. A non-dominated sorting genetic algorithm is used to solve the multi-objective optimization problem.

6. The node role allocation method in distributed training according to claim 3, characterized in that, The optimization of the node role allocation scheme based on the training performance data includes: Compare each performance metric in the training performance data with the corresponding preset threshold; Determine the target performance index that deviates from the preset threshold; Based on the degree of deviation between the target performance index and the preset threshold, the node role allocation scheme is optimized; the optimized node role allocation scheme includes the reselection of service nodes or worker nodes, or the redistribution of the affiliation relationship between worker nodes and service nodes.

7. The node role allocation method in distributed training according to claim 1, characterized in that, The method further includes: Periodically re-examine the computing resource information of the computing devices in the cluster; The computing network resource map is updated using the computing resource information of the computing devices in the cluster that have been re-explored.

8. A node role allocation device in distributed training, characterized in that, include: A construction module is used to construct a computing power network resource graph based on the computing power resource information of computing power devices in the cluster; wherein, the nodes of the computing power network resource graph represent the computing power devices, and the attributes of the nodes include resource status; the edges of the computing power network resource graph represent the network connections between the computing power devices, and the attributes of the edges include communication performance. The generation module is used to generate a node role allocation scheme for the distributed training task to be executed based on the computing power network resource map. The allocation module is used to allocate service node roles or worker node roles to each computing device in the cluster based on the node role allocation scheme, and to start the distributed training task. The monitoring module is used to monitor training performance data during distributed training. The optimization module is used to optimize the node role allocation scheme based on the training performance data; The reassignment module is used to reassign node roles based on the optimized node role allocation scheme.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being coupled to the memory; the memory is used to store computer instructions, which are loaded and executed by the processor to enable the computer device to implement the node role assignment method in distributed training as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer-executable instructions that, when executed on a computer, cause the computer to perform the node role allocation method in distributed training as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when run on an electronic device, causes the electronic device to perform the node role allocation method in distributed training as described in any one of claims 1 to 7.

Images