Distributed training parallel strategy search method, device and system

Abstract

The invention relates to a distributed training parallel strategy search method, device and system. The searching method comprises the following steps: collecting hardware resource information X (i) and network communication performance information Y (i) of each computing node i; extracting static element features of the target model and configuration parameter features of the parallel strategy; searching an applicable parallel strategy set K according to the hardware resource information X (i) of each computing node i and the static meta-feature of the target model; according to the hardware resource information X (i) and the network communication performance information Y (i) of each computing node i, the static meta-feature of the target model and the configuration parameter feature of the applicable parallel strategy set K, calculating the training performance index of the target model which is trained by executing the parallel strategy at the computing node i; and according to an optimization target specified by a user, screening and executing the most adaptive parallel strategy through training performance indexes. The distributed training parallel strategy search method provided by the invention has the advantages that global optimal parallel strategy selection can be realized, and the training effect is good.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a distributed training parallel policy search method, apparatus and system. Background Technology

[0002] With the explosive growth of model parameters, the computing power of a single node is no longer sufficient to meet the needs of training large models. Distributed training, as a core technology for achieving multi-node collaborative computing, involves various parallel strategies, including data parallelism, tensor parallelism, pipelined parallelism, and hybrid parallelism. These strategies exhibit significant differences in training performance under different hardware configurations, network topologies, and computational task characteristics.

[0003] Currently, distributed training relies heavily on human experience in selecting parallel strategies. It requires users to manually design parallel schemes based on a deep understanding of various resource characteristics (such as GPU computing power, memory bandwidth, network bandwidth and communication latency). This has a high professional threshold and lacks the ability to jointly perceive and model the overall state of the system, making it difficult to achieve the globally optimal selection of parallel strategies. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide a distributed training parallel policy search method, apparatus and system.

[0005] A distributed training parallel policy search method includes: S10. Collect hardware resource information X(i) and network communication performance information Y(i) of each computing node i. S20. Extract the static meta-features of the target model and the configuration parameter features of the parallel strategy; S30. Based on the hardware resource information X(i) of each computing node i and the static meta-features of the target model, search for the set of applicable parallel strategies K. S40. Based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i, the static meta-features of the target model, and the configuration parameter features of the applicable parallel strategy set K, calculate the parallel strategy to be executed on computing node i. Training performance metrics for the target model. ; S50. Based on the optimization goal specified by the user, select and execute the most suitable parallel strategy through training performance indicators.

[0006] The distributed training parallel strategy search method described in this invention collects hardware resource information and network communication performance information of computing nodes, and extracts static meta-features of the target model and configuration parameter features of the parallel strategy respectively. Then, the three are concatenated and fused through a T-GNN graph neural network prediction model to obtain a joint representation vector that can characterize the overall state of the system. Based on the joint representation vector, the training throughput y1, single-step communication time y2, and peak memory usage y3 of each parallel strategy are calculated for training the target model. Finally, the most suitable parallel strategy is selected and executed according to the optimization objective, thereby achieving the globally optimal parallel strategy selection. The entire process is automatic and requires no manual settings, reducing the control threshold.

[0007] Further, step S40 includes: S41. Construct a resource graph G based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i; S42. Obtain the feature vector of the target model based on the static meta-features of the target model. ; S43. Based on each parallel strategy in the set of applicable parallel strategies K. Based on the configuration parameter characteristics, obtain the parallel strategies. eigenvectors ; S44, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors Perform feature fusion to obtain a joint representation vector. ; S45. Based on the joint representation vector The target model is calculated based on various parallel strategies. Training throughput at compute node i Single-step communication time and peak memory usage .

[0008] Furthermore, the distributed training parallel strategy search method also includes the following steps: S60, periodically calculate the performance error LOSS of the target model training, and if the performance error LOSS exceeds the threshold, repeat steps S10 to S50.

[0009] Further, step S41 includes: S411, Construct node set V; S412, Construct edge set E; S413. Construct the feature matrix of node set V. ; S414. Construct the feature matrix of edge set E. ; S415, Construct resource graph G.

[0010] Further, step S44 includes: S441, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors The continuous numerical features in the data are normalized. S442. Aggregate information from adjacent nodes to obtain information from each node. Local topology-aware vectors of adjacent nodes ; S443. Employ an attention mechanism to weighted aggregate the contributions of different neighboring nodes, applying this to all nodes. Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. ; S444, Represent the global graph as a vector With the normalized target model feature vector and various parallel strategies eigenvectors By splicing and merging, a joint representation vector is obtained. .

[0011] Furthermore, the specific process of step S442 is as follows: For each node Perform L-layer convolution iterative calculations. After the L-th convolution calculation, each node... Local topology-aware vectors of adjacent nodes It can be calculated using the following formula:

[0012] In the formula, For nodes The input value computed in the Lth convolution layer, For each node initial feature vector N(i) is a node The set of adjacent nodes, For connecting nodes With neighboring nodes edge eigenvectors, For edge feature fusion module, and The learning parameters are calculated for the Lth convolutional layer. It is a non-linear activation function. This is a vector concatenation operation.

[0013] Furthermore, the specific process of step S443 is as follows: compute nodes Local topology-aware vectors of adjacent nodes Attention weights :

[0014] For each node Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. :

[0015] In the formula, To determine the feature vector based on the parallel strategy The generated context query vector, Here are the learnable parameters, and m is the number of nodes. The total number.

[0016] Furthermore, the specific process of step S444 is as follows: The joint representation vector is calculated using the following formula. :

[0017] In the formula, For fusion function, This is a vector concatenation operation.

[0018] A distributed training parallel policy search device includes: The multi-dimensional resource acquisition module is used to collect hardware resource information X(i) and network communication performance information Y(i) of each computing node i. The feature extraction module is used to extract the static meta-features of the target model and the configuration parameter features of the parallel strategy; The strategy search module is used to search for a set of applicable parallel strategies K based on the hardware resource information X(i) of each computing node i and the static meta-features of the target model. The performance prediction module is used to calculate the parallel strategy to be executed on computing node i based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i, the static meta-features of the target model, and the configuration parameter features of the applicable parallel strategy set K. Training performance metrics for the target model. ; The execution module is used to select and execute the most suitable parallel strategy based on the user-specified optimization goal and training performance metrics.

[0019] A distributed training parallel policy search system includes: at least one computing node i and the distributed training parallel policy search device as described in claim 9; The computing node i includes at least one host and one device for training the target model; the distributed training parallel strategy search device searches for and executes the most suitable parallel strategy for training the target model on the computing node i based on the feature information of the target model and each computing node i, and according to the optimization objective specified by the user.

[0020] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0021] Figure 1 The execution flowchart of the distributed training parallel policy search method provided by the present invention; Figure 2 The execution flowchart of step S40 provided by the present invention; Figure 3 The execution flowchart of step S41 provided by the present invention; Figure 4 The execution flowchart of step S44 provided by the present invention; Figure 5 This is a schematic diagram of the distributed training parallel policy search device provided by the present invention. Detailed Implementation

[0022] This invention provides a distributed training parallel strategy search method, apparatus, and system based on graph neural network technology. First, it collects hardware resource information and network communication performance information of computing nodes, and then extracts the static meta-features of the target model and the configuration parameter features of the parallel strategy. Based on the above information, it constructs a resource graph G and a target model feature vector, respectively. and the eigenvectors of each parallel strategy A T-GNN graph neural network prediction model is jointly constructed. The three components are then concatenated and fused using the T-GNN prediction model to obtain a joint representation vector that characterizes the overall system state. Based on this joint representation vector, the training throughput of the target model trained using each parallel strategy is calculated. Single-step communication time Peak memory usage Finally, based on the optimization objective, the most suitable parallel strategy is selected and executed, thereby achieving the globally optimal parallel strategy selection. Furthermore, by periodically calculating the training error of the target model, and reselecting the most suitable parallel strategy when the training error exceeds a threshold, the system's real-time perception and dynamic response to performance fluctuations during training can be improved, ensuring the stability of the training task.

[0023] Please refer to Figures 1 to 5 , Figure 1 The execution flowchart of the distributed training parallel policy search method provided by the present invention; Figure 2 The execution flowchart of step S40 provided by the present invention; Figure 3 The execution flowchart of step S41 provided by the present invention; Figure 4 The execution flowchart of step S44 provided by the present invention; Figure 5 This is a schematic diagram of the distributed training parallel policy search device provided by the present invention.

[0024] This invention provides a distributed training parallel strategy search system, comprising at least one computing node i and a distributed training parallel strategy search device. The computing node i includes at least one host and one device for training a target model. The distributed training parallel strategy search device, based on the target model and the feature information of each computing node i, and according to the optimization objective specified by the user, searches for and executes the parallel strategy most suitable for training the target model on the computing node i.

[0025] The distributed training parallel policy search device includes a multi-dimensional resource acquisition module 10, a feature extraction module 20, a policy search module 30, a performance prediction module 40, an execution module 50, and a feedback module 60.

[0026] The multi-dimensional resource acquisition module 10 is used to perform step S10: acquire hardware resource information X(i) and network communication performance information Y(i) of each computing node i.

[0027] In this embodiment, a lightweight agent program continuously collects hardware resource information X(i) and network communication performance information Y(i) of each computing node i at a period of 1 to 3 seconds.

[0028] Among them, the hardware resource information X(i) includes GPU type X1(i), number of GPU cards X2(i), video memory capacity X3(i), computing power X4(i), CPU load X5(i) and storage I / O performance X6(i).

[0029] The network communication performance information Y(i) includes the inter-node communication bandwidth Y1(i), communication delay Y2(i), packet loss rate Y3(i), and topological distance Y4(i).

[0030] The feature extraction module 20 is used to perform step S20: extracting the static meta-features of the target model and the configuration parameter features of the parallel strategy. Specifically, step S20 includes: S21. Extract the static meta-features of the target model.

[0031] In this embodiment, static meta-features of the target model are extracted through analytical functions, including the parameter size, inter-layer dependencies, computational density, and whether the target model contains long sequences or sparse structures.

[0032] S22. Extract the configuration parameter features of the parallel strategy.

[0033] In this embodiment, the configuration parameter features of the parallel strategy are extracted by the parsing function, including data parallelism DP, tensor parallelism TP, pipeline parallelism PP, sequence parallelism enable flag SP, parameter sharding strategy type H, activation checkpoint enable flag C, and data format used for training FP.

[0034] The strategy search module 30 is used to perform step S30: based on the hardware resource information X(i) of each computing node i and the static meta-features of the target model, search for an applicable set of parallel strategies K.

[0035] In this embodiment, all are first enumerated using an algorithm. The parallel strategy combination is N, which is the sum of the number of GPU cards X2(i) of each computing node i, ensuring that the computing resources required by the parallel strategy do not exceed the available scale of all computing nodes i.

[0036] Then, semantic constraints are applied based on the structural features of the target model: (1) If the target model does not contain an operator structure that supports tensor partitioning, then exclude all tensor parallelism. Parallel strategy; (2) If there is no sequential dependency between layers of the target model, then exclude all pipeline parallelism. Parallel strategy.

[0037] Finally, based on the parameter size of the target model and the data format FP used for training, a fast memory usage estimation is performed. Parallel strategies whose peak memory requirements exceed the memory capacity of each computing node i x 3(i) are excluded, resulting in a set of applicable parallel strategies K. , where n is the total number of applicable parallel strategies.

[0038] The performance prediction module 40 is used to execute step S40: based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i, the static meta-features of the target model, and the configuration parameter features of the applicable parallel strategy set K, calculate the parallel strategy to be executed on computing node i. Training performance metrics for the target model. .

[0039] Specifically, the training performance metrics of the target model include training throughput. Single-step communication time Peak memory usage In this embodiment, the above performance metrics are calculated by constructing a graph neural network prediction model that jointly models "resources-models-strategies". Step S40 includes: S41. Construct a resource graph G based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i. Specifically, step S41 includes: S411, Construct node set V.

[0040] In this embodiment, each computing node i is mapped to a node in the resource graph. Define all compute nodes The set is the node set V. m is the computing node The total number.

[0041] S412, Construct edge set E.

[0042] In this embodiment, the physical network connections between computing nodes are mapped to edges of the resource graph. , Represents the i-th node in node set E. and the j-th node The connection relationship between them.

[0043] S413. Construct the feature matrix of node set V. .

[0044] In this embodiment, a feature vector is defined based on the hardware resource information X(i) of each computing node i. =[X1(i), X2(i), X3(i), X4(i), X5(i), X6(i)], for each node Assign the corresponding feature vector Put all nodes eigenvectors Stack them in order to form a feature matrix. .

[0045] S414. Construct the feature matrix of edge set E. .

[0046] In this embodiment, a feature vector is defined based on the network communication performance information Y(i) of each computing node i. =[Y1(i), Y2(i), Y3(i), Y4(i)], for each edge Assign the corresponding feature vector , put all the edges eigenvectors Stack them in order to form a feature matrix. .

[0047] S415, construct resource graph G.

[0048] In this embodiment, a resource graph G = (V, E, ...) is constructed. , In this context, node set E describes the location information of each computing node i in the resource graph, and edge set E describes the physical network connections between computing nodes. Describe each node in the node set E Feature information, Describe each edge in the edge set E. The characteristic information.

[0049] S42. Obtain the feature vector of the target model based on the static meta-features of the target model. .

[0050] In this embodiment, the target model feature vector is defined. =[ , , , , , ].

[0051] in, The parameter scale corresponding to the target model can be encoded and calculated using logarithmic scaling. and Corresponding to inter-layer dependencies, which describe the topology of the target model's computation graph, two fundamental features are obtained based on the computation graph: the graph diameter and the average path length. These features are used as... and Input; The computational density corresponding to the target model can be calculated using the analysis tool PyTorch Profiler; and Whether the target model contains long sequences or sparse structures is determined through binary encoding. This indicates that the target model does not contain long sequences. This indicates that the target model contains long sequences. This indicates that the target model does not have a sparse structure. This indicates that the target model has a sparse structure.

[0052] S43. Based on each parallel strategy in the set of applicable parallel strategies K. Based on the configuration parameter characteristics, obtain the parallel strategies. eigenvectors .

[0053] In this embodiment, each parallel strategy is defined. eigenvectors =[ , , , , , , ].

[0054] in, The corresponding data parallelism is DP, where DP represents the number of data partitions, expressed as an integer. The corresponding tensor parallelism TP, where TP represents the dimension of the model tensor being split, expressed as an integer; The corresponding pipeline parallelism PP represents the number of pipeline stages into which the model layer is divided, expressed as an integer; The corresponding sequence parallelism enable flag SP indicates whether sequence parallelism is enabled. SP is represented in binary code, with 1 indicating that sequence parallelism is enabled and 0 indicating that sequence parallelism is not enabled. The corresponding parameter is the fragmentation strategy type H, which includes five modes: no fragmentation, ZeRO level 1 to 3, and FSDP. It is represented by integer encoding, with 0 representing no fragmentation, 1 representing ZeRO-1, 2 representing ZeRO-2, 3 representing ZeRO-3, and 4 representing FSDP. The corresponding activation flag C indicates whether the checkpointing technology is activated. It is represented by binary code, with 1 indicating that the checkpointing technology is activated and 0 indicating that the checkpointing technology is not activated. The corresponding training data format FP, including FP16 and FP32, is represented by integer encoding, with 0 representing FP16 and 1 representing FP32.

[0055] S44, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors Perform feature fusion to obtain a joint representation vector. .

[0056] In this embodiment, a three-layer cascaded graph convolutional network structure is used to process the resource graph G and the target model feature vector. and various parallel strategies eigenvectors Deep integration is performed to capture hardware resource dependencies, network communication bottlenecks, and "model-resource-policy" adaptability. Specifically, step S34 includes: S441, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors The continuous numerical features in the data are normalized.

[0057] Specifically, the first-layer cascaded graph convolutional network uses Z-Score normalization to normalize the resource graph G and the target model feature vector. and characteristics of each parallel strategy vector The continuous numerical features in the dataset are normalized to eliminate their dimensions and improve computational accuracy. These continuous numerical features include memory capacity X3(i), computational power X4(i), communication bandwidth Y1(i), communication latency Y2(i), parameter size, computational density, data parallelism DP, tensor parallelism TP, and pipeline parallelism PP. The normalized standard values ​​are calculated using the following formula:

[0058] In the formula, x is the original eigenvalue. These are the normalized eigenvalues. and denoted as the mean and standard deviation of the original feature value x, respectively.

[0059] S442. Aggregate information from adjacent nodes to obtain information from each node. Local topology-aware vectors of adjacent nodes .

[0060] Specifically, the second-layer cascaded graph convolutional network aggregates information from adjacent nodes through multi-layer graph convolutional iterative computation, and performs this computation on each node. Its own feature vector The feature vectors of all neighboring nodes and their corresponding edges eigenvectors Merge to obtain each node By associating the local topology sensing vectors of adjacent nodes, the local network topology and potential communication bottlenecks can be perceived.

[0061] Assuming the second-layer cascaded graph convolutional network performs L layers of convolutional iterations, then after the Lth convolutional layer computation, each node... Local topology-aware vectors of adjacent nodes It can be calculated using the following formula:

[0062] In the formula, For nodes The input value computed in the Lth convolution layer, For each node initial feature vector N(i) is a node The set of adjacent nodes, For connecting nodes With neighboring nodes edge eigenvectors, For edge feature fusion module, and The learning parameters are calculated for the Lth convolutional layer. It is a non-linear activation function. This is a vector concatenation operation.

[0063] S443. Employ an attention mechanism to weighted aggregate the contributions of different neighboring nodes, applying this to all nodes. Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. .

[0064] Specifically, the second-layer cascaded graph convolutional network obtains the value of each node through multi-layer graph convolutional iterative computation. Local topology-aware vectors of adjacent nodes The third-layer cascaded graph convolutional network is for each node. Local topology-aware vectors of adjacent nodes Assign attention weights Then, a weighted sum is performed to obtain the global graph representation vector. .

[0065] Each node Local topology-aware vectors of adjacent nodes Attention weights Calculated using the following formula:

[0066] In the formula, To determine the feature vector based on the parallel strategy The generated context query vector, Here are the learnable parameters, and m is the number of nodes. The total number.

[0067] For each node Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. :

[0068] S444, Represent the global graph as a vector With the normalized target model feature vector and various parallel strategies eigenvectors By splicing and merging, a joint representation vector is obtained. .

[0069] In this embodiment, the global graph representation vector is... With the normalized target model feature vector Parallel strategy feature vectors The vectors are spliced ​​and merged to form a unified joint representation vector. The joint representation vector It fully integrates cluster resource topology, model computation characteristics, and policy configuration semantics, providing high-order feature inputs for performance prediction. Joint representation vector The calculation formula is:

[0070] In the formula, For fusion function, This is a vector concatenation operation.

[0071] S45. Based on the joint representation vector The target model is calculated based on various parallel strategies. Training throughput at compute node i Single-step communication time Peak memory usage .

[0072] In this embodiment, a multi-task learning framework is adopted to jointly represent vectors. As input, through a multilayer perceptron network The target model is calculated based on various parallel strategies. Training throughput at compute node i Single-step communication time Peak memory usage :

[0073] The execution module 50 is used to execute step S50: according to the optimization target specified by the user, the most suitable parallel strategy is selected and executed by training performance indicators.

[0074] Specifically, the system selects training throughput corresponding to the user-specified optimization objectives, such as fastest training, lowest communication overhead, or lowest GPU memory usage. Maximum, single-step communication time Minimum or peak video memory usage Minimal parallel strategy, and its corresponding feature vector. The configuration parameters are converted into those that the deep learning framework PyTorch can recognize, and the corresponding startup script, distributed process group partitioning scheme, communication backend settings, pipeline stage boundaries and activation checkpoint strategies are automatically generated, so that the parallel strategy can be applied to train the target model.

[0075] The feedback module 60 is used to perform step S60: periodically calculate the performance error LOSS of the target model training. If the performance error LOSS exceeds the threshold, re-execute steps S10 to S50.

[0076] In this embodiment, during the training of the target model according to the recommended parallel strategy, the performance error (LOSS) between the current training performance of the target model and the predicted training performance of the target model is calculated in 30-second intervals using the following formula:

[0077] In the formula, , , Training throughput , , The corresponding task weight, This represents the measured training throughput during target model training. This represents the measured value of the single-step communication time during target model training. This represents the measured peak memory usage during target model training.

[0078] The system sets a performance error threshold based on the training accuracy requirements. The system compares the performance error LOSS with the performance error threshold. When the performance error LOSS > the performance error threshold, the optimization mechanism is triggered, and steps S10 to S50 are re-executed to select the parallel strategy most suitable for the current training situation.

[0079] This invention provides a distributed training parallel strategy search method, apparatus, and system based on graph neural network technology. It obtains a joint representation vector that characterizes the overall system state by concatenating and fusing hardware resource information and network communication performance information of computing nodes, static meta-features of the target model, and configuration parameter features of parallel strategies. The training throughput of the target model trained using each parallel strategy is then calculated based on the joint representation vector. Single-step communication time Peak memory usage Finally, based on the optimization objective, the most suitable parallel strategy is selected and executed, thereby achieving globally optimal parallel strategy selection. The entire process is automated, requiring no manual configuration and reducing control requirements. Furthermore, by periodically calculating the training error of the target model, and reselecting the most suitable parallel strategy when the training error exceeds a threshold, the system's real-time perception and dynamic response to performance fluctuations during training are improved, ensuring the stability of the training task.

[0080] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a distributed training parallel policy search method and system as described in any of the above embodiments.

[0081] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and the present invention also intends to include these modifications and variations.

Claims

1. A distributed training parallel policy search method, characterized in that, include: S10. Collect hardware resource information X(i) and network communication performance information Y(i) of each computing node i. S20. Extract the static meta-features of the target model and the configuration parameter features of the parallel strategy; S30. Based on the hardware resource information X(i) of each computing node i and the static meta-features of the target model, search for the set of applicable parallel strategies K. S40. Based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i, the static meta-features of the target model, and the configuration parameter features of the applicable parallel strategy set K, calculate the parallel strategy to be executed on computing node i. Training performance metrics for the target model. ; S50. Based on the optimization goal specified by the user, select and execute the most suitable parallel strategy through training performance indicators.

2. The distributed training parallel policy search method according to claim 1, characterized in that, Step S40 includes: S41. Construct a resource graph G based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i; S42. Obtain the feature vector of the target model based on the static meta-features of the target model. ; S43. Based on each parallel strategy in the set of applicable parallel strategies K. Based on the configuration parameter characteristics, obtain the parallel strategies. eigenvectors ; S44, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors Perform feature fusion to obtain a joint representation vector. ; S45. Based on the joint representation vector The target model is calculated based on various parallel strategies. Training throughput at compute node i Single-step communication time Peak memory usage .

3. The distributed training parallel policy search method according to claim 1, characterized in that, Also includes: S60. Periodically calculate the performance error LOSS of the target model training. If the performance error LOSS exceeds the threshold, repeat steps S10 to S50.

4. The distributed training parallel policy search method according to claim 2, characterized in that, Step S41 includes: S411, Construct node set V; S412, Construct edge set E; S413. Construct the feature matrix of node set V. ; S414. Construct the feature matrix of edge set E. ; S415, Construct resource graph G.

5. The distributed training parallel policy search method according to claim 2, characterized in that, Step S44 includes: S441, Regarding the resource graph G and the feature vector of the target model and various parallel strategies eigenvectors The continuous numerical features in the data are normalized. S442. Aggregate information from adjacent nodes to obtain information from each node. Local topology-aware vectors of adjacent nodes ; S443. Employ an attention mechanism to weighted aggregate the contributions of different neighboring nodes, applying this to all nodes. Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. ; S444, Represent the global graph as a vector With the normalized target model feature vector and various parallel strategies eigenvectors By splicing and merging, a joint representation vector is obtained. .

6. The distributed training parallel policy search method according to claim 5, characterized in that, The specific process of step S442 is as follows: For each node Perform L-layer convolution iterative calculations. After the L-th convolution calculation, each node... Local topology-aware vectors of adjacent nodes It can be calculated using the following formula: In the formula, For nodes The input value computed in the Lth convolution layer, For each node initial feature vector N(i) is a node The set of adjacent nodes, For connecting nodes With neighboring nodes edge eigenvectors, For edge feature fusion module, and The learning parameters are calculated for the Lth convolutional layer. It is a non-linear activation function. This is a vector concatenation operation.

7. The distributed training parallel policy search method according to claim 6, characterized in that, The specific process of step S443 is as follows: compute nodes Local topology-aware vectors of adjacent nodes Attention weights : For each node Local topology-aware vectors of adjacent nodes Perform a weighted summation to obtain the global graph representation vector. : In the formula, To determine the feature vector based on the parallel strategy The generated context query vector, Here are the learnable parameters, and m is the number of nodes. The total number.

8. The distributed training parallel policy search method according to claim 7, characterized in that, The specific process of step S444 is as follows: The joint representation vector is calculated using the following formula. : In the formula, For fusion function, This is a vector concatenation operation.

9. A distributed training parallel policy search device, characterized in that, include: The multi-dimensional resource acquisition module is used to collect hardware resource information X(i) and network communication performance information Y(i) of each computing node i. The feature extraction module is used to extract the static meta-features of the target model and the configuration parameter features of the parallel strategy; The strategy search module is used to search for a set of applicable parallel strategies K based on the hardware resource information X(i) of each computing node i and the static meta-features of the target model. The performance prediction module is used to calculate the parallel strategy to be executed on computing node i based on the hardware resource information X(i) and network communication performance information Y(i) of each computing node i, the static meta-features of the target model, and the configuration parameter features of the applicable parallel strategy set K. Training performance metrics for the target model. ; The execution module is used to select and execute the most suitable parallel strategy based on the user-specified optimization goal and training performance metrics.

10. A distributed training parallel policy search system, characterized in that, include: At least one computing node i and the distributed training parallel policy search device as described in claim 9; The computing node i includes at least one host and one device for training the target model; the distributed training parallel strategy search device searches for and executes the most suitable parallel strategy for training the target model on the computing node i based on the feature information of the target model and each computing node i, and according to the optimization objective specified by the user.

Images