Geographically distributed training node selection method, system, storage medium and device

Abstract

This application provides a method, system, storage medium, and device for node selection in geographically distributed training. The method includes: acquiring a distributed training task and available nodes; forming a candidate node set by assembling nodes with computing power-carbon emission efficiency that meet the requirements, and adding the nodes to the initial training set in descending order of computing power-carbon emission efficiency; calculating the initial gradient aggregation nodes, the initial carbon emission amount for a single training round, and the initial completion time of the initial training set; the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set; replacing the slow node with an iterative node and calculating the iterative carbon emission amount; if the iterative carbon emission amount is less than the carbon emission amount before the iteration, generating a new iterative training set; repeatedly replacing the slow node with an iterative node that meets the carbon emission requirements until the number of iterations reaches a preset maximum number of iterations. This solves the problems of low synchronization efficiency and poor overall carbon emission performance in the distributed training process.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to node selection methods, systems, storage media, and devices for geographically distributed training. Background Technology

[0002] With the explosive growth of artificial intelligence technology, the number of model parameters has reached trillions, and the training process requires massive computing and power resources. In distributed training, multiple computing nodes need to collaborate to complete iterative calculations under a unified scheduling mechanism. As the training model continues to expand, the energy consumption and carbon emission problems under the existing scheduling mechanism are becoming increasingly prominent, restricting the sustainable operation of geographically distributed training.

[0003] Due to significant differences in energy structures across different geographical regions, the real-time carbon intensity of data center nodes exhibits marked spatiotemporal heterogeneity. Existing green scheduling strategies typically aim to reduce carbon emissions per unit of computing power, prioritizing data centers with lower carbon intensity for training. However, these strategies often rely on static or instantaneous carbon intensity indicators at the node level, failing to adequately consider the synchronous execution characteristics of distributed model training. Distributed model training generally employs a synchronous iterative mechanism, requiring all participating nodes to complete local computation and gradient communication before proceeding to the next round. When node selection is based solely on low carbon intensity, nodes with weaker computational performance or higher communication latency are easily introduced, becoming slow nodes in the training process. Since servers still consume significant basic energy even when idle, the ineffective idle carbon emissions caused by high-performance nodes synchronously waiting for slow nodes often offset or even exceed the emission reduction benefits of cross-regional scheduling to low-carbon nodes. Therefore, it is crucial to consider both the spatiotemporal heterogeneity of node carbon intensity and the synchronization efficiency and overall carbon emission performance during distributed training. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a node selection method, system, storage medium and device for geographically distributed training, which solves the problems of low synchronization efficiency and poor overall carbon emission effect in the distributed training process.

[0005] To achieve one of the aforementioned objectives, this application provides a node selection method for geographically distributed training, the method comprising:

[0006] Obtain distributed training tasks and available nodes, wherein the nodes have fixed computing power and carbon emission intensity; Nodes whose computing power-carbon emission efficiency meets the requirements are grouped into a candidate node set, and a predetermined number of these nodes are added to the initial training set in descending order of computing power-carbon emission efficiency; wherein, computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity. Calculate the initial gradient aggregation node, the initial carbon emissions for a single training round, and the initial completion time in the initial training set; wherein, the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set; The slow node is replaced with an iterative node, which is recorded as one iteration. The iterative gradient aggregation node, iteration completion time, and iteration carbon emissions are calculated after the iteration. The iterative node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but not to the initial training set. If the carbon emissions in the iteration are less than the carbon emissions before the iteration, a new iterative training set is generated; The slow node is replaced multiple times by the iterative node that meets the carbon emission requirements until the preset maximum number of iterations is reached, and then the iterative training set and its iterative gradient aggregation node are output.

[0007] As a further improvement to one embodiment of this application, the calculation of the initial gradient aggregation node includes: Traverse the initial training set and calculate the average communication delay between each node in the initial training set and other nodes, wherein the average communication delay includes the average propagation delay and the average transmission delay; The node with the smallest average communication latency in the initial training set is selected as the initial gradient aggregation node.

[0008] As a further improvement to one embodiment of this application, the calculation of the initial completion time in the initial training set includes: Calculate the training time and communication time for each node in the initial training set, where the completion time of a node is the sum of the training time and the communication time. The node with the longest completion time is selected as the slow node.

[0009] As a further improvement to one embodiment of this application, the calculation of the initial carbon emissions for a single round of training in the initial training set includes: Calculate the idle time of the nodes in the initial training set, excluding the slow node; The initial carbon emissions are calculated using the following formula: ; in, The initial carbon emissions for a single round of training. The time spent by the node during the training phase. This refers to the power consumption of the server during training. The communication time of the node. The idle time of the node. CI represents the base power consumption of the server during communication and idling, and CI represents carbon emission intensity.

[0010] As a further improvement to one embodiment of this application, the calculation of the iterative gradient aggregation node, iteration completion time, and iteration carbon emissions after iteration includes: If the carbon emissions of the iteration are greater than or equal to the carbon emissions before the iteration, then the iteration node used for replacement is deleted from the iteration training set, and the iteration training set of the previous iteration or the initial training set is retained.

[0011] As a further improvement to one embodiment of this application, the step of forming a candidate node set from nodes that meet the requirements for computing power-carbon emission efficiency includes: Calculate the ratio of computing power to carbon emission intensity for each node to generate the computing power-carbon emission efficiency of the node; An efficiency threshold is set, and nodes whose computing power-carbon emission efficiency is greater than or equal to the efficiency threshold are combined into the candidate node set.

[0012] As a further improvement to one embodiment of this application, before selecting a preset number of the most efficient nodes from the candidate node set to add to the initial training set, the process includes: Set a training set and a gradient aggregation node, wherein the training set and the gradient aggregation node are empty sets; The step of replacing the slow node with an iterative node includes: The slow node is removed from the initial training set or the iterative training set, and the iterative node is added to the iterative training set.

[0013] Based on the same inventive concept, this application also provides a node selection system for geographically distributed training, comprising: An acquisition module is used to acquire distributed training tasks and available nodes, wherein the nodes include fixed computing power and carbon emission intensity; The aggregation module is used to form a candidate node set by assembling nodes that meet the requirements of computing power-carbon emission efficiency, and to add a preset number of the nodes in the candidate node set to the initial training set in descending order of computing power-carbon emission efficiency; wherein, the computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity; The calculation module is used to calculate the initial gradient aggregation node, the initial carbon emissions of a single training round, and the initial completion time in the initial training set; wherein, the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set; The iteration module is used to replace the slow node with the iteration node, which is recorded as one iteration and calculates the iteration gradient aggregation node, iteration completion time and iteration carbon emissions after the iteration; wherein, the iteration node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but do not belong to the initial training set. The judgment module is used to generate a new iterative training set if the carbon emissions in the iteration are less than the carbon emissions before the iteration. The output module is used to repeatedly replace the slow node with the iterative node that meets the carbon emission requirements until the number of iterations reaches the preset maximum number of iterations, and output the iterative training set and its iterative gradient aggregation node.

[0014] Based on the same inventive concept, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to execute the above-described node selection method for geographically distributed training.

[0015] Based on the same inventive concept, this application also provides an electronic device, including: a processor and a memory; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the node selection method for geographic distributed training.

[0016] Compared with the prior art, the technical effect of the present invention is as follows: taking into account the synchronous execution characteristics of distributed training, the computing performance of computing nodes, communication latency and carbon emission intensity, etc., under the premise of meeting the synchronous requirements of model training, the set of nodes participating in training is determined to reduce the ineffective energy consumption and carbon emissions caused by synchronization waiting during training, thereby improving the overall carbon efficiency of distributed training and effectively reducing the overall carbon emissions of the training process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the implementation methods or related technologies will be briefly introduced below. Obviously, the drawings described below are only the implementation methods of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of geographic distributed training based on the PS architecture in the existing technology; Figure 2A flowchart illustrating a node selection method for geographic distributed training provided in one embodiment of this application; Figure 3 A flowchart illustrating a node selection method for geographic distributed training, provided as another embodiment of this application; Figure 4 A node topology diagram provided for another specific embodiment of this application; Figure 5 A schematic diagram of a node selection system for geographically distributed training provided in another embodiment of this application; Figure 6 This is a schematic diagram of the hardware structure of an electronic device provided for another embodiment of this application. Detailed Implementation

[0019] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0020] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by those skilled in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects.

[0021] The technical background of this invention is geographic distributed training based on a PS (Parameter Server) architecture, such as... Figure 1 As shown. Distributed training accelerates the training process by performing model computations in parallel across multiple nodes. Typically, distributed training is conducted in iterations, with each iteration including stages such as local computation ①, gradient uploading ②, gradient aggregation ③, and parameter distribution ④.

[0022] In the local computation phase ①, each computing node performs forward and backward computations of the model based on its own data copy, generating corresponding gradient results. The time required to complete local computation depends on the computing hardware performance and available computing resources of the nodes. In the gradient uploading phase ②, each computing node needs to send the locally generated gradients to the aggregation node via the network. This phase involves cross-node data communication, and its transmission latency is affected by factors such as network link conditions and node geographical location. In the gradient aggregation phase ③, after receiving the gradients from all relevant nodes, the aggregation node summarizes the gradient information and updates the model parameters. In the parameter distribution phase ④, the updated model parameters are sent to each computing node, and each node can only enter the next round of training after receiving the new parameters.

[0023] In existing technologies, to reduce carbon emissions during geographically distributed training, computational tasks are typically deployed to computing nodes with lower carbon emissions. However, geographically distributed training usually employs a synchronous update mechanism, requiring all participating nodes to complete local computation and gradient transfer before parameter aggregation and updates can occur in each training round. When carbon emissions are used as the primary criterion for node selection, it is easy to select nodes with lower computational performance or cause other nodes to wait for extended periods during the synchronization phase. Since computing nodes continue to consume basic energy during the waiting period, the ineffective idle energy consumption and carbon emissions caused by synchronization waiting often offset or even exceed the emission reduction benefits gained by introducing low-carbon nodes, making it difficult to effectively improve the overall carbon efficiency of geographically distributed training.

[0024] To address the aforementioned problems, this application provides a node selection method for geographically distributed training, such as... Figure 2 As shown, the method includes the following steps: Step S100: Obtain the distributed training task and available nodes, where the nodes include fixed computing power and carbon emission intensity.

[0025] Specifically, computing power and carbon emission intensity are characteristics of each node. In subsequent steps, appropriate nodes are selected based on each node's computing power and carbon emission intensity to maximize the carbon efficiency of the entire distributed training process. Distributed tasks also include the number of nodes required to execute the task and the computational load of the task.

[0026] In one possible implementation of this application, the method prior to step S200 includes: Step S110: Set the training set and gradient aggregation node, where the training set and gradient aggregation node are empty sets.

[0027] Step S200: Nodes whose computing power-carbon emission efficiency meets the requirements are grouped into a candidate node set, and a preset number of nodes are added to the initial training set in the candidate node set in descending order of computing power-carbon emission efficiency; wherein, computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity.

[0028] Specifically, the computing power carbon emission efficiency of the nodes in the candidate node set is sorted from high to low, and the top k nodes are selected to form the initial training set.

[0029] It should be noted that the preset number is the number of nodes required for distributed training.

[0030] In one possible implementation of this application, step S200 further includes: Step S210: Calculate the ratio of computing power to carbon emission intensity for each node to generate the computing power-carbon emission efficiency of the node.

[0031] Step S220: Set an efficiency threshold and combine nodes whose computing power-carbon emission efficiency is greater than or equal to the efficiency threshold into a candidate node set.

[0032] Specifically, the computing power carbon emission efficiency index is defined as node computing power (TFLOPS) / node carbon emission intensity (kgCO2e / kWh), and the computing power-carbon emission efficiency of each node is calculated.

[0033] Step S300: Calculate the initial gradient aggregation node, the initial carbon emissions for a single training round, and the initial completion time in the initial training set; wherein, the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set.

[0034] In one possible implementation of this application, calculating the initial gradient aggregation node in step S300 includes: Step S311: Traverse the initial training set and calculate the average communication delay between each node and other nodes in the initial training set. The average communication delay includes the average propagation delay and the average transmission delay.

[0035] Step S312: Select the node with the smallest average communication latency from the initial training set as the initial gradient aggregation node.

[0036] Specifically, the average communication delay is denoted as The gradient aggregation node is denoted as P.

[0037] In one possible implementation of this application, calculating the initial completion time in step S300 includes: Step S321: Calculate the training time and communication time of each node in the initial training set. The completion time of a node is the sum of the training time and communication time.

[0038] Step S322: Select the node with the longest completion time as the slow node.

[0039] In one possible implementation of this application, calculating the initial carbon emissions for a single round of training in step S300 includes: Step S331: Calculate the idle time of nodes other than slow nodes in the initial training set.

[0040] Step S332, calculate the initial carbon emissions according to the following formula: ; in, This represents the initial carbon emissions for a single training round. This refers to the time a node spends during the training phase. This refers to the power consumption of the server during training. This refers to the communication time of the nodes. This refers to the idle time of the node. CI represents the server's base power consumption during communication and idling, and C represents carbon emission intensity.

[0041] Specifically, the carbon emissions of a single training round consist of the carbon emissions from the calculation phase and the synchronization waiting phase. The total electricity consumed in one training session is converted into carbon emissions by multiplying the total electricity consumption by the carbon emissions.

[0042] Step S400: Replace the slow node with the iterative node, record it as one iteration, and calculate the iterative gradient aggregation node, iteration completion time, and iteration carbon emissions after the iteration; where the iterative node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but not to the initial training set.

[0043] Specifically, the nodes in the initial training set can achieve maximum training efficiency, but it is still necessary to iterate on the nodes in the training set to find a balance between high training efficiency and low carbon emissions, so as to maximize carbon utilization. Slow nodes are replaced by nodes that were not selected in the initial training set but belong to the candidate node set.

[0044] In step S500, if the carbon emissions in the iteration are less than the carbon emissions before the iteration, a new iterative training set is generated.

[0045] In one possible implementation of this application, the method further includes the following after step S400: Step S510: If the carbon emissions of the iteration are greater than or equal to the carbon emissions before the iteration, then delete the iteration node used for replacement in the iteration training set, and retain the iteration training set of the previous iteration or the initial training set.

[0046] Step S600: Repeatedly replace the slow node with the iterative node that meets the carbon emission requirements until the number of iterations reaches the preset maximum number of iterations, and output the iterative training set and its iterative gradient aggregation node.

[0047] Specifically, when the initial training set has not been iterated, the iteration count is recorded as 0. Subsequent replacements will increment the iteration count by 1 until the maximum iteration count is reached, completing the iteration and outputting the final iteration result.

[0048] In another implementation of this application, such as Figure 3 As shown, another node selection method for geographically distributed training is provided, including: Step S1: The distributed model training task arrives. The training task attributes include the number of computing nodes k required for this round of training and the computational cost w of the task.

[0049] Step S2: Initialize the training node set S and gradient aggregation node P as empty sets.

[0050] Step S3: Set the maximum number of iteration rounds M, and initialize the current number of iteration rounds m=0.

[0051] Step S4: Obtain computing node information and calculate its computing power-carbon emission efficiency index: For any computing node i, obtain its node status information, including the computing power parameter Ci and the carbon emission intensity Ri. Define the computing power-carbon emission efficiency index = node computing power (TFLOPS) / node carbon emission intensity (kgCO2e / kWh), and calculate the computing power-carbon emission efficiency of each node.

[0052] Step S5: Set an efficiency threshold, remove nodes whose computing power-carbon emission efficiency is lower than the threshold, and use the remaining nodes as the candidate node set N.

[0053] Step S6: Sort the nodes in the candidate node set N from high to low according to computing power-carbon emission efficiency, and select the top k nodes to form the initial training set S.

[0054] Step S7: Traverse the k nodes and calculate the average communication delay t from each node to other training nodes. Communication delay includes propagation delay and transmission delay. The node with the minimum average communication delay is selected as the gradient aggregation node P.

[0055] Step S8: For each node i, calculate its training time. and communication time The synchronization completion time of each node The node with the longest synchronization completion time is the slowest node, and the overall synchronization completion time is... .

[0056] Step S9: For each non-slow node i, calculate its idle time. The carbon emissions of a single training round consist of the carbon emissions from the computation phase and the synchronization waiting phase. The power consumption of the computation phase is the power consumption of the server during training. The power consumption during the synchronization waiting phase is the basic power consumption generated by the server during communication and idling waiting. Calculate the carbon emissions of a single training round.

[0057] Step S10, update the iteration count m←m+1.

[0058] Step S11: Select the node with the highest computing power carbon emission efficiency from the candidate node set N, excluding the current training node set S, and use it to replace the slow node identified in set S, thereby constructing a new S′.

[0059] Step S12: Repeat steps S7-S9 to calculate the gradient aggregation node P′ and the total carbon emissions E′ of the new set.

[0060] Step S13: Determine whether the carbon emissions E′ of set S′ are less than the carbon emissions E of set S. If yes, proceed to step S14; otherwise, proceed to step S15.

[0061] Step S14: Update S=S′, P=P′, E=E′. When the total carbon emissions generated by the new set are smaller, update the training node set S, the gradient aggregation node P, and the total carbon emissions E.

[0062] Step S15: Determine whether the preset number of iterations has been reached. If yes, proceed to step S17; otherwise, proceed to step S16.

[0063] Step S16: Delete the nodes added in the previous round from the candidate training node set N to avoid repeatedly using nodes that have been verified as suboptimal.

[0064] Step S17: When the preset number of iterations is reached, the iteration stops, and the training node set S and gradient aggregation node P at this time are output.

[0065] Step S18, the entire process is complete.

[0066] Another implementation of this application provides an alternative node selection method for geographically distributed training, including: A carbon efficiency-aware training node selection method for geographically distributed training can be applied to geographically distributed model training scenarios. Specific implementation examples are described below. Figure 4 As shown, the data centers are interconnected via a wide area network. Table 1 below displays information related to the computing power, carbon emission intensity, and power consumption of each node.

[0067] Table 1 - Node Parameter Table:

[0068] Suppose a distributed training task arrives, requiring 4 training nodes (k=4). The computation cost of a single round (local iteration) of a node is 100 TFLOPs. After every 100 iterations of local training, a global gradient aggregation is performed, and the model gradient size is 500MB.

[0069] Initialize the training node set S and gradient aggregation node P as empty sets. Set the maximum number of iterations per node M=2, and initialize the number of iterations m=0.

[0070] For each node i in the topology, its computing power-carbon emission efficiency is calculated as {250, 250, 200, 310, 320, 500, 312.5, 200, 220}. Nodes with a computing power-carbon emission efficiency lower than 250 are removed, resulting in a candidate node set N = {1, 2, 4, 5, 6, 7}. The nodes in set N are sorted in descending order of computing power-carbon emission efficiency, and the first four nodes are selected to form the initial training set S = {2, 5, 6, 7}. When these four nodes in the set are used as gradient aggregation nodes, the average communication latency to other nodes is as follows: =(4.008+4.014+4.012) / 3=4.011, =(4.008+4.004+4.017) / 3=4.0097, =(4.014+4.017+4.013) / 3=4.0147, =(4.018+4.004+4.013) / 3=4.0117. Select node 5, which has the smallest average communication latency, as the gradient aggregation node P. The synchronization completion times for each node are then: t2=50+4.008=54.008, t5=125, t6=200+4.017=204.017, t7=80+4.004=84.004, and the overall synchronization completion time T= =204.017, Node 6 is the slowest node in this group, with an initial carbon emission of E≈0.283kgCO2.

[0071] The update iteration count is m=1. Node 4 replaces the slow node 6 in the training set S, resulting in a new training node set S′={2, 4, 5, 7}. When the four nodes in this set are used as gradient aggregation nodes, the average communication latency to other nodes is as follows: =(4.005+4.008+4.012) / 3=4.008, =(4.008+4.003+4.004) / 3=4.005, =(4.005 + 4.003 + 4.007) / 3 = 4.005, =(4.007 + 4.004 + 4.012) / 3 = 4.011. Select node 4 with the minimum average communication delay as the gradient aggregation node P'. At this time, the synchronization completion time of each node is: t2 = 50 + 4.005 = 54.005, t5 = 125 + 4.003 = 129.003, t4 = 80.65, t7 = 80 + 4.007 = 84.004, and the overall synchronization completion time T = = 129.003. Node 5 is the slow node in this group of nodes, and the initial carbon emission is E'≈0.2234 kgCO2. Since E' < E, update the training node set S = {2, 4, 5, 7}, the gradient aggregation node P = {4}, and the overall carbon emission E≈0.2234 kgCO2.

[0072] Since m≠M, update the iteration number m = 2. Replace the slow node 5 in the training set S with node 1 to obtain the new training node set S' = {1, 2, 4, 7}. When the four nodes in the set are used as the gradient aggregation nodes respectively, the average communication delays to other nodes are: =(4.004 + 4.004 + 4.011) / 3 = 4.0063, =(4.008 + 4.004 + 4.012) / 3 = 4.008, =(4.005 + 4.004 + 4.007) / 3 = 4.0053, =(4.011 + 4.007 + 4.012) / 3 = 4.01. Select node 4 with the minimum average communication delay as the gradient aggregation node P'. At this time, the synchronization completion time of each node is: t1 = 200 + 4.003 = 204.003, t2 = 50 + 4.005 = 54.005, t4 = 80.65, t7 = 80 + 4.007 = 84.004, and the overall synchronization completion time T = = 204.003. Node 1 is the slow node in this group of nodes, and the initial carbon emission is E'≈0.286 kgCO2. Since E' > E, do not update the parameters.

[0073] At this time, the number of node replacements m = M. Output S = {2, 4, 5, 7}, P = {4}. Select data centers 2, 4, 5, 7 as the training node set, select data center 4 as the gradient aggregation node, and the process ends.

[0074] In another embodiment of the present application, a node selection system for geographically distributed training is disclosed, as Figure 5 shown, including: The acquisition module is used to acquire distributed training tasks and available nodes, which include fixed computing power and carbon emission intensity. The set module is used to group nodes that meet the requirements of computing power-carbon emission efficiency into a candidate node set, and add a preset number of nodes from the candidate node set in descending order of computing power-carbon emission efficiency to the initial training set; where computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity. The calculation module is used to calculate the initial gradient aggregation node in the initial training set, the initial carbon emissions for a single round of training, and the initial completion time; where the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set. The iteration module is used to replace slow nodes with iterative nodes, which is recorded as one iteration and calculates the iterative gradient aggregation node, iteration completion time and iteration carbon emissions after the iteration. Among them, the iterative node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but not to the initial training set. The judgment module is used to generate a new iterative training set if the carbon emissions in the iteration are less than the carbon emissions before the iteration. The output module is used to repeatedly replace slow nodes with iterative nodes that meet carbon emission requirements until the preset maximum number of iterations is reached, and outputs the iterative training set and its iterative gradient aggregation node.

[0075] Figure 6 This diagram illustrates a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are interconnected internally via the bus 1050.

[0076] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0077] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0078] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0079] The communication interface 1040 is used to connect the communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, radio (shortwave / ultra-shortwave) communication, satellite communication, data link communication, etc.).

[0080] Bus 1050 includes pathways for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0081] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments described in this specification, and need not include all the components shown in the figures.

[0082] The electronic device described above is used to implement the node selection method for geographically distributed training in any of the foregoing embodiments, and has the beneficial effects of the corresponding method implementation, which will not be elaborated here.

[0083] Based on the same inventive concept, corresponding to any of the above-described embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the node selection method for geographic distributed training as described in any of the above embodiments.

[0084] The computer-readable medium in this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0085] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the node selection method for geographically distributed training as described in any of the above embodiments, and have the beneficial effects of the corresponding method implementations, which will not be repeated here.

[0086] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; this manner of description is merely for clarity, and those skilled in the art should consider the specification as a whole. Within the framework of this application, the above embodiments or the technical features of different embodiments can also be appropriately combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0087] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be entirely within the understanding of those skilled in the art). While specific details (e.g., circuits) are set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0088] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0089] The embodiments described herein are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and principles of the embodiments described herein should be included within the protection scope of this application.

Claims

1. A method for selecting a node for geo-distributed training, the method comprising: The method includes: Obtain distributed training tasks and available nodes, wherein the nodes have fixed computing power and carbon emission intensity; Nodes whose computing power-carbon emission efficiency meets the requirements are grouped into a candidate node set, and a predetermined number of these nodes are added to the initial training set in descending order of computing power-carbon emission efficiency; wherein, computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity. Calculate the initial gradient aggregation node, the initial carbon emissions for a single training round, and the initial completion time in the initial training set; wherein, the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set; The slow node is replaced with an iterative node, which is recorded as one iteration. The iterative gradient aggregation node, iteration completion time, and iteration carbon emissions are calculated after the iteration. The iterative node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but not to the initial training set. If the carbon emissions in the iteration are less than the carbon emissions before the iteration, a new iterative training set is generated; The slow node is replaced multiple times by the iterative node that meets the carbon emission requirements until the preset maximum number of iterations is reached, and then the iterative training set and its iterative gradient aggregation node are output.

2. The node selection method for geographically distributed training according to claim 1, characterized in that, The calculation of the initial gradient aggregation node includes: Traverse the initial training set and calculate the average communication delay between each node in the initial training set and other nodes, wherein the average communication delay includes the average propagation delay and the average transmission delay; The node with the smallest average communication latency in the initial training set is selected as the initial gradient aggregation node.

3. The node selection method for geographically distributed training according to claim 1, characterized in that, The calculation of the initial completion time in the initial training set includes: Calculate the training time and communication time for each node in the initial training set, where the completion time of a node is the sum of the training time and the communication time. The node with the longest completion time is selected as the slow node.

4. The node selection method for geographically distributed training according to claim 1, characterized in that, The calculation of the initial carbon emissions for a single round of training in the initial training set includes: Calculate the idle time of the nodes in the initial training set, excluding the slow node; The initial carbon emissions are calculated using the following formula: ； in, The initial carbon emissions for a single round of training. The time spent by the node during the training phase. This refers to the power consumption of the server during training. The communication time of the node. The idle time of the node. CI represents the base power consumption of the server during communication and idling, and CI represents carbon emission intensity.

5. The node selection method for geographically distributed training according to claim 1, characterized in that, The calculation of the iterative gradient aggregation node, iteration completion time, and iterative carbon emissions after iteration includes: If the carbon emissions of the iteration are greater than or equal to the carbon emissions before the iteration, then the iteration node used for replacement is deleted from the iteration training set, and the iteration training set of the previous iteration or the initial training set is retained.

6. The node selection method for geographically distributed training according to claim 1, characterized in that, The process of forming a candidate node set from nodes that meet the requirements for computing power-carbon emission efficiency includes: Calculate the ratio of computing power to carbon emission intensity for each node to generate the computing power-carbon emission efficiency of the node; An efficiency threshold is set, and nodes whose computing power-carbon emission efficiency is greater than or equal to the efficiency threshold are combined into the candidate node set.

7. The node selection method for geographically distributed training according to claim 1, characterized in that, Before selecting a preset number of the most efficient nodes from the candidate node set to add to the initial training set, the process includes: Set a training set and a gradient aggregation node, wherein the training set and the gradient aggregation node are empty sets; The step of replacing the slow node with an iterative node includes: The slow node is removed from the initial training set or the iterative training set, and the iterative node is added to the iterative training set.

8. A node selection system for geographically distributed training, characterized in that, The system includes: An acquisition module is used to acquire distributed training tasks and available nodes, wherein the nodes include fixed computing power and carbon emission intensity; The aggregation module is used to form a candidate node set by assembling nodes that meet the requirements of computing power-carbon emission efficiency, and to add a preset number of the nodes in the candidate node set to the initial training set in descending order of computing power-carbon emission efficiency; wherein, the computing power-carbon emission efficiency is the ratio of computing power to carbon emission intensity; The calculation module is used to calculate the initial gradient aggregation node, the initial carbon emissions of a single training round, and the initial completion time in the initial training set; wherein, the initial completion time is the completion time of the slowest node with the longest completion time in the initial training set; The iteration module is used to replace the slow node with the iteration node, which is recorded as one iteration and calculates the iteration gradient aggregation node, iteration completion time and iteration carbon emissions after the iteration; wherein, the iteration node is the node with the highest computing power-carbon emission efficiency among the nodes that belong to the candidate node set but do not belong to the initial training set. The judgment module is used to generate a new iterative training set if the carbon emissions in the iteration are less than the carbon emissions before the iteration. The output module is used to repeatedly replace the slow node with the iterative node that meets the carbon emission requirements until the number of iterations reaches the preset maximum number of iterations, and output the iterative training set and its iterative gradient aggregation node.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions for causing the computer to execute the node selection method of any one of claims 1 to 7 for geographic distributed training.

10. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the node selection method for geographic distributed training as described in any one of claims 1 to 7.

Images