Model distributed training efficiency test system based on multiple machines and multiple cards

Abstract

The invention provides a model distributed training efficiency test system based on multiple machines and multiple cards. The system comprises a test management middle table, a multi-machine multi-card environment construction module, a test demand matching module, a strategy matching module, a model training test module and a training efficiency evaluation module. The multi-machine multi-card environment construction module constructs a multi-machine multi-card environment under different network environments; the test demand matching module performs matching to obtain a target model and a target data set; a strategy matching module performs matching to obtain a target parallel strategy combination; the model training test module performs distributed training by using the target parallel strategy combination and the target data set based on each multi-machine multi-card environment to obtain index data in each multi-machine multi-card environment; and the training efficiency evaluation module evaluates the influence result of the target parallel strategy combination on the training efficiency of the target model in the multi-machine multi-card environment of each network environment based on the index data. According to the invention, the evaluation accuracy of the training efficiency of the model in distributed training is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a model distributed training efficiency testing system based on multiple machines and multiple cards. Background Art

[0002] In the field of deep learning, as model size continues to grow and datasets become increasingly massive, the computing resources of a single machine and a single graphics card can no longer meet training requirements. Therefore, distributed training of models has become crucial.

[0003] The primary current method for testing the efficiency of distributed model training involves setting up a multi-machine, multi-GPU environment in a relatively fixed and ideal network environment. This involves pre-setting stable bandwidth, extremely low and constant latency, and near-zero packet loss. During testing, a specific model is selected from a fixed model library and data from a pre-defined dataset. This specific model is then tested within this multi-machine, multi-GPU environment using a pre-defined single parallel strategy.

[0004] However, in actual application scenarios, different models and datasets may be suitable for different parallel strategies. A single fixed parallel strategy cannot fully utilize the computing potential of a multi-machine, multi-GPU environment. In addition, complex and changeable network conditions may lead to low resource utilization, making it impossible to accurately evaluate the true efficiency of the model in distributed training. On the other hand, in actual application scenarios, different bandwidths, delays, and packet loss rates will have a significant impact on the efficiency of distributed model training. Existing methods are unable to simulate these complex conditions, making their test environments seriously disconnected from the complex network conditions in actual application scenarios. As a result, the tested training efficiency does not truly reflect the performance of the model in actual applications, which may cause deviations in the model during distributed training, thus affecting the accurate evaluation of the model's training efficiency in distributed training. Summary of the Invention

[0005] The present invention provides a model distributed training efficiency testing system based on multiple machines and multiple cards, aiming to improve the evaluation accuracy of the training efficiency of the model in distributed training.

[0006] In a first aspect, the present invention provides a model distributed training efficiency testing system based on multiple machines and multiple cards, comprising a test management center, a multi-machine and multi-card environment construction module, a test requirement matching module, a strategy matching module, a model training test module, and a training efficiency evaluation module; the test management center is respectively connected to the multi-machine and multi-card environment construction module, the test requirement matching module, the strategy matching module, the model training test module, and the training efficiency evaluation module to manage each module;

[0007] The multi-machine and multi-card environment construction module is used to simulate the network based on the node layout and network parameters between each node in the multi-machine and multi-card system, and to build a multi-machine and multi-card environment under different network environments;

[0008] The test requirement matching module is used to match the model library and the dataset warehouse according to the test requirements to obtain the target model to be tested and the target dataset for testing;

[0009] A strategy matching module, configured to perform matching in a parallel strategy library based on the model structure characteristics of the target model and the data scale characteristics of the target data set to obtain a target parallel strategy combination;

[0010] A model training and testing module is used to perform distributed training on the target model using the target parallel strategy combination and the target data set in a multi-machine and multi-card environment with different network environments, and obtain indicator data of the target model in each multi-machine and multi-card environment during the training process;

[0011] The training efficiency evaluation module is used to evaluate the indicator data in each multi-machine and multi-card environment to obtain the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment.

[0012] In a second aspect, the present invention further provides a method for testing the efficiency of distributed model training based on multiple machines and multiple cards, which is implemented based on the system for testing the efficiency of distributed model training based on multiple machines and multiple cards described in the first aspect. The method for testing the efficiency of distributed model training based on multiple machines and multiple cards includes:

[0013] Conduct network simulation based on the node layout and network parameters between nodes in a multi-machine and multi-card system, and build a multi-machine and multi-card environment under different network conditions;

[0014] According to the test requirements, the model library and the dataset warehouse are matched to obtain the target model to be tested and the target dataset for testing;

[0015] Matching the model structure characteristics of the target model and the data scale characteristics of the target data set in a parallel strategy library to obtain a target parallel strategy combination;

[0016] Performing distributed training on the target model using the target parallel strategy combination and the target data set in a multi-machine and multi-card environment with different network environments, and obtaining indicator data of the target model in each multi-machine and multi-card environment during the training process;

[0017] Based on the indicator data in each multi-machine and multi-card environment, the evaluation is performed to obtain the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment.

[0018] In a third aspect, the present invention also provides an electronic device, comprising: a memory for storing a computer software program; a processor for reading and executing the computer software program, thereby implementing the model distributed training efficiency testing method based on multiple machines and multiple cards as described above.

[0019] In a fourth aspect, the present invention also provides a non-transitory computer-readable storage medium, in which a computer software program is stored. When the computer software program is executed by a processor, the distributed training efficiency testing method of the model based on multiple machines and multiple cards as described above is implemented.

[0020] In a fifth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned model distributed training efficiency testing method based on multiple machines and multiple cards.

[0021] The embodiment of the present invention provides a model distributed training efficiency test system based on multiple machines and multiple cards. It performs network simulation through the node layout and network parameters between each node in the multi-machine and multi-card system, and constructs a multi-machine and multi-card environment under different network environments. Therefore, it simulates the network parameters under different environments in actual application scenarios, so that the test environment of the model is more consistent with the complex network conditions in the actual application scenario, and avoids the problem that the deviation caused by the disconnection between the test environment and the actual application scenario affects the evaluation accuracy of the model training efficiency. On the other hand, according to the model structure characteristics of the target model and the data scale characteristics of the target data set, the corresponding parallel strategy combination is matched in the parallel strategy library. Therefore, the parallel strategy can be dynamically selected for different models and data sets, avoiding the problem that the low resource utilization caused by a single fixed parallel strategy affects the evaluation accuracy of the model training efficiency. Therefore, the embodiment of the present invention improves the evaluation accuracy of the training efficiency of the model in distributed training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a structural diagram of a multi-machine and multi-GPU model distributed training efficiency testing system provided by the present invention;

[0023] Figure 2 This is a flow chart of the multi-machine and multi-GPU model distributed training efficiency testing method provided by the present invention;

[0024] Figure 3 An embodiment diagram of an electronic device provided by an embodiment of the present invention;

[0025] Figure 4 An embodiment diagram of a computer-readable storage medium provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0027] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the specified features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0028] In the description of the present invention, the term "for example" is used to mean "used as an example, illustration or illustration". Any embodiment of the present invention described as "for example" is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes are not elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed herein.

[0029] Optional, see Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of the multi-machine, multi-card model distributed training efficiency testing system provided by the present invention. The multi-machine, multi-card model distributed training efficiency testing system includes a test management center, a multi-machine, multi-card environment construction module, a test requirement matching module, a strategy matching module, a model training test module, and a training efficiency evaluation module. The test management center in the embodiment of the present invention is connected to the multi-machine, multi-card environment construction module, the test requirement matching module, the strategy matching module, the model training test module, and the training efficiency evaluation module, respectively, to manage each module.

[0030] Optionally, the multi-machine multi-card system in an embodiment of the present invention includes multiple computing nodes, each computing node is such as a CPU, GPU, etc. Therefore, the multi-machine multi-card environment construction module obtains the node layout and network parameters between each node in the multi-machine multi-card system, wherein the node layout covers information such as the physical location and topological structure of the node, and the network parameters include bandwidth, delay, packet loss rate, etc.

[0031] Furthermore, the multi-machine multi-card environment construction module uses a network simulation tool to perform network simulation based on the node layout of the multi-machine multi-card system and the network parameters between nodes, and constructs a multi-machine multi-card environment under different network conditions, as described in steps 101 to 104 below. In one embodiment, the multi-machine multi-card system includes three computing nodes, node 1 and node 2 are connected through a network with a bandwidth of 10Gbps, a delay of 1ms, and a packet loss rate of 0.1%; node 2 and node 3 are connected through a network with a bandwidth of 25Gbps, a delay of 0.5ms, and a packet loss rate of 0.05%. The multi-machine multi-card environment construction module uses a network simulation tool (such as ns-3) to construct a network topology based on these parameters. Nodes are defined in ns-3, and attributes such as bandwidth, delay, and packet loss rate of the links between nodes are set, thereby constructing a multi-machine multi-card network environment that meets the above parameters.

[0032] Optionally, embodiments of the present invention pre-establish mapping relationships between test requirements, models in a model library, and datasets in a dataset warehouse. The model library stores models of various types and structures, while the dataset warehouse stores datasets of various fields and sizes. Therefore, the test requirement matching module obtains test requirements, such as testing the performance of a model on a large dataset or testing the effectiveness of a specific structural model on a certain type of dataset.

[0033] Furthermore, the test requirement matching module matches the model library and the dataset warehouse according to the test requirements in combination with the mapping relationship, and obtains the target model to be tested and the target dataset for testing. In one embodiment, the test requirement is to evaluate the performance of a deep learning model for image classification on a large-scale image dataset. The model library contains a variety of image classification models such as AlexNet, VGG16, and ResNet50, and the dataset warehouse contains image datasets such as CIFAR-10, CIFAR-100, and ImageNet. The test requirement matching module selects ResNet50 as the target model in the model library based on the test requirements and the mapping relationship, because it performs well in large-scale image classification tasks; and selects ImageNet as the target dataset in the dataset warehouse because it is large in scale and covers a rich range of image categories, which meets the test requirements for large-scale datasets.

[0034] Optionally, the parallel strategy library in the embodiment of the present invention stores a plurality of parallel strategies targeting the model structure characteristics of different models and the data scale characteristics of data sets.

[0035] Therefore, the strategy matching module obtains the model structure characteristics of the target model (such as the number of model layers, the number of model parameters) and the data scale characteristics of the target data set (the number of data samples, the data dimension, etc.), and matches them in the parallel strategy library based on the model structure characteristics of the target model and the data scale characteristics of the target data set. By comparing the characteristics of the target model and the data set with the applicable conditions of each strategy in the parallel strategy library, the most suitable target parallel strategy combination is found, as described in steps 301 to 304.

[0036] Optionally, the model training test module performs distributed training on the target model in a multi-machine and multi-card environment in different network environments using a target parallel strategy combination combined with a target data set. During the training process, various indicator data of the target model in each multi-machine and multi-card environment are recorded, as specifically described in steps 401 to 404, including indicator data such as average delay time, transmission rate fluctuation, training progress balance, and computing resource utilization (CPU utilization, GPU utilization), etc.

[0037] Optionally, the training efficiency evaluation module calculates the comprehensive index coefficient in each multi-machine and multi-card environment based on the average delay time, transmission rate fluctuation, training progress balance and computing resource utilization in each multi-machine and multi-card environment. The comprehensive index coefficient can be calculated by a weighted algorithm. In one embodiment, each indicator data is assigned a corresponding weight and weighted summed to obtain the comprehensive index coefficient.

[0038] Furthermore, the training efficiency evaluation module evaluates the impact of the parallel strategy combination in each multi-machine and multi-card environment on the training efficiency of the target model based on the comprehensive index coefficient in each multi-machine and multi-card environment. By comparing the comprehensive index coefficients of the parallel strategy combinations in different multi-machine and multi-card environments, conclusions are drawn as to which parallel strategy combination can enable the target model to achieve the best training efficiency in the multi-machine and multi-card environment.

[0039] The embodiment of the present invention simulates the network by analyzing the node layout and network parameters between each node in a multi-machine multi-card system, and constructs a multi-machine multi-card environment under different network environments. This simulates the network parameters under different environments in actual application scenarios, making the model test environment more consistent with the complex network conditions in actual application scenarios, and avoiding the problem of the deviation caused by the disconnection between the test environment and the actual application scenario affecting the evaluation accuracy of the model training efficiency. On the other hand, the corresponding parallel strategy combination is matched in the parallel strategy library according to the model structure characteristics of the target model and the data scale characteristics of the target data set. Therefore, the parallel strategy can be dynamically selected for different models and data sets, avoiding the problem of low resource utilization caused by a single fixed parallel strategy affecting the evaluation accuracy of the model training efficiency, thereby improving the evaluation accuracy of the model training efficiency in distributed training.

[0040] In one embodiment, steps 101 to 104 are described as follows:

[0041] Step 101: abstract the node layout of the multi-machine and multi-card system into a multi-dimensional space model.

[0042] Optionally, to accurately process the spatial relationships of nodes in a multi-machine, multi-card system, the test system converts the node layout into a multidimensional spatial model, where the position coordinates of each node correspond to its actual location in the real physical environment. In one embodiment, there are three nodes in the multi-machine, multi-card system: A, B, and C. In the three-dimensional spatial model, the position coordinates of node A are set to (1, 2, 3), the position coordinates of node B are set to (4, 5, 6), and the position coordinates of node C are set to (7, 8, 9).

[0043] Step 102: construct a distance matrix between nodes based on the position coordinates of each node in the multidimensional space model, and generate an initial network topology based on the distance matrix.

[0044] Furthermore, the test system calculates the distance between any two nodes based on the position coordinates of the nodes in the multidimensional space model using the two-point distance formula to construct a distance matrix. Therefore, each matrix element in the distance matrix represents the distance between the corresponding two nodes.

[0045] Continuing with the above embodiment, based on the position coordinates of nodes A, B, and C combined with the distance formula, the distance between nodes A and B is obtained as The distance between node A and node C is The distance between node B and node C is Therefore, the distance matrix D is as follows:

[0046]

[0047] Furthermore, the test system screens the distance matrix according to a preset distance threshold. When the matrix elements corresponding to two first target nodes in the matrix are less than or equal to the preset distance threshold, a connection relationship between the two first target nodes is established, and the matrix elements between any two nodes in the distance matrix are traversed to generate an initial network topology. Therefore, it can be understood that the matrix elements in the distance matrix of the two first target nodes with a connection relationship in the initial network topology are less than or equal to the preset distance threshold.

[0048] Continuing with the above embodiment, the preset distance threshold is 5. Since the distance between node A and node B, the distance between node A and node C, and the distance between node B and node C are all greater than 5, in the generated initial network topology, there is a connection relationship between node A and node B, a connection relationship between node A and node C, and a connection relationship between node B and node C.

[0049] Step 103 : adjusting the initial edge weight of the first target node in the initial network topology based on the network bandwidth between the first target nodes to obtain an adjusted network topology.

[0050] Optionally, in the initial network topology, the initial edge weight of the first target node with a connection relationship in the initial network topology is 1. Therefore, the test system obtains the network bandwidth between the first target nodes and calculates the adjustment coefficient of the edge weight between the first target nodes based on the network bandwidth between the first target nodes. The specific formula is: w ij =log 10 (B ij +1), where w ij represents the adjustment coefficient of the edge weight between node i and node j in the first target node, B ij represents the network bandwidth between node i and node j.

[0051] Furthermore, the test system adjusts the initial edge weight of the first target node in the initial network topology by multiplying the adjustment coefficient of the edge weight between the first target nodes and the initial edge weight, and updates the initial network topology according to the adjusted edge weight of the first target node to obtain the adjusted network topology.

[0052] In one embodiment, in the initial network topology, there are connected nodes A and B, and B and C. The network bandwidth between A and B is 8 Gbps, and the network bandwidth between B and C is 10 Gbps. Therefore, the adjustment coefficient of the edge weight between A and B is log 10 (8+1)=0.954, the adjustment coefficient of the edge weight between node B and node C is log 10 (10+1)=1.041. Therefore, the adjusted edge weight of node A and node B is 0.954, and the adjusted edge weight of node B and node C is 1.041.

[0053] Step 104 : Perform network simulation based on the adjusted network topology in combination with the network delay and the network packet loss rate between the first target nodes to build a multi-machine and multi-card environment under different network environments.

[0054] Furthermore, the test system performs network simulation based on the adjusted network topology combined with the network delay and network packet loss rate between the first target nodes to construct a multi-machine and multi-card environment under different network environments, as described in steps 1041 to 1044.

[0055] The embodiment of the present invention constructs a multi-machine and multi-card environment under different network environments according to the location of the nodes in the multi-machine and multi-card system, the network bandwidth, the network delay and the network packet loss rate, thereby simulating the network parameters in different environments in the actual application scenarios, so that the test environment of the model is more in line with the complex network conditions in the actual application scenarios, avoiding the problem that the deviation caused by the disconnection between the test environment and the actual application scenario affects the evaluation accuracy of the model training efficiency, and improving the evaluation accuracy of the model training efficiency in distributed training.

[0056] In one embodiment, steps 1041 to 1044 are described as follows:

[0057] Step 1041 : For any two second target nodes in the adjusted network topology, an optimal intermediate node sequence of the second target nodes in the adjusted network topology is planned based on the shortest path algorithm and the adjusted edge weights of the first target nodes.

[0058] Optionally, in the adjusted network topology, the test system applies a shortest path algorithm to two arbitrarily selected second target nodes, combining the adjusted edge weights of the first target node to plan an optimal path. In this embodiment of the present invention, the shortest path is not based solely on distance, but rather comprehensively considers the data transmission capacity represented by the edge weights. Therefore, the shortest path algorithm can find the path from one second target node to another with the highest data transmission efficiency under the current network topology. The sequence of intermediate nodes traversed by this path is the optimal sequence of intermediate nodes.

[0059] Optionally, an embodiment of the present invention adopts the Dijkstra algorithm as the shortest path algorithm, and the adjusted network topology is G = (V, E), where V is a node set and E is an edge set. For any two second target nodes, the source node s and the target node t, their distance array d[v] represents the shortest path length from the source node s to the node v. Initially, d[s] = 0. For other nodes v≠s, d[v] = +∞. The set S is used to store nodes for which the shortest path has been determined. In each iteration, the node u closest to the source node is selected from the nodes not in the set S, and is added to the set S. Then, the distance d[v] = min(d[v], d[u] + w(u, v) of the node v adjacent to the node u is updated, where w(u, v) is the edge weight of the edge (u, v). This process is repeated until the target node t is added to the set S. At this time, the shortest path from the source node s to the target node t, that is, the optimal intermediate node sequence, can be obtained by backtracing the d array.

[0060] In one embodiment, in the adjusted network topology, the second target nodes are nodes D and E. The network topology contains nodes A, B, C, D, and E, and the edges and corresponding weights are as follows: (A, B, 2), (A, C, 3), (B, C, 1), (B, D, 4), (C, E, 5), and (D, E, 3). Node D is used as the source node, and the Dijkstra algorithm is applied. Initially, d[D] = 0, and the distances to other nodes are +∞. In the first step, the only node adjacent to node D is node B, and d[B] = 4 is updated. Next, among the nodes whose shortest paths have not been determined, node B is closest to node D, and node B is added to set S. The nodes adjacent to node B are then updated, d[A] = min(+∞, 4+2) = 6, and d[C] = min(+∞, 4+1) = 5. Afterwards, node C is closest to node D, so node C is added to set S, and d[E] = min(+∞, 5+5) = 10 is updated. Finally, the shortest path from node D to node E is determined to be D→B→C→E, and the optimal intermediate node sequence is obtained.

[0061] Step 1042: Determine the comprehensive network environment parameters of the second target node based on the network delay, network packet loss rate and adjusted back-edge weights between the nodes in the optimal intermediate node sequence.

[0062] Furthermore, the test system comprehensively considers the network delay, network packet loss rate and subsequent weight adjustment between each adjacent node in the sequence based on the optimal intermediate node sequence, and determines the comprehensive network environment parameters between the two second target nodes, as described in steps 10421 to 10423.

[0063] Step 1043 : Determine the transmission rate of the second target node based on the comprehensive network environment parameters of the second target node, the first processing capability of the sending node, and the second processing capability of the receiving node.

[0064] Furthermore, the test system uses the comprehensive network environment parameters of the second target node, and combines the first processing capability of the sending node (the amount of data that can be processed per unit time) and the second processing capability of the receiving node to calculate the transmission rate between the second target nodes through the transmission rate formula, wherein the transmission rate formula in the embodiment of the present invention is:

[0065] R=min(C s ,C r ) / [1+L*W*(1+P)].

[0066] Among them, R represents the transmission rate, C s Indicates the first processing capability of the sending node, C r represents the secondary processing capability of the receiving node, L represents the overall network latency, P represents the overall network packet loss rate, and W represents the overall edge weight. 1+L*W*(1+P) represents the network environment's barrier to transmission rate. Longer network latency, larger edge weights, and higher packet loss rates increase the barrier factor and lower the transmission rate.

[0067] In one embodiment, the first processing capability C of the sending node s =100Mbps, the second processing capacity of the receiving node C r =120Mbps, the comprehensive network environment parameters are (9ms, 0.6%, 10), the transmission rate is:

[0068] R=min(100,120) / [1+0.009*10*(1+0.006)]≈91.7Mbps.

[0069] Step 1044 , simulating the network transmission process of data in the multi-machine multi-card system at the transmission rate of the second target node, and constructing a multi-machine multi-card environment under different network environments.

[0070] Furthermore, the test system simulates the network transmission process of data in a multi-machine, multi-card system based on the transmission rate of the second target node. By varying the selection of different second target node pairs, as well as the network topology changes (such as node additions, edge weight readjustments, etc.) and network latency and packet loss rate parameters that may occur in steps 1041 to 1043, a variety of multi-machine, multi-card environments can be constructed under different network conditions. Each network environment can reflect the impact of different network conditions on data transmission, thus providing a rich variety of network simulation scenarios for model training and testing.

[0071] In one embodiment, different second target node pairs are selected, such as node F and node G, and their transmission rates are recalculated according to steps 1041 to 1043. In the new network topology, the optimal intermediate node sequence of node F and node G is different from before, and the network delay, packet loss rate and edge weight have also changed, and the transmission rate is calculated to be 70Mbps. Then node H and node I are selected again, and different transmission rates are obtained, such as 110Mbps. By continuously selecting different node pairs and calculating the transmission rates, combined with changes in network topology and parameters, a multi-machine and multi-card environment under a variety of different network environments is constructed. For example, in one environment, the transmission rate between some nodes is fast and the network is stable (low packet loss rate, low delay), and in another environment, the transmission rate between some nodes is slow and the network is unstable, thereby simulating different network conditions that may occur in actual applications.

[0072] The embodiment of the present invention constructs different network environments that can comprehensively reflect various complex network conditions in a multi-machine and multi-card system, covering different transmission rates, network delays, packet loss rates and connection relationships between nodes, so that the test environment of the model is more in line with the complex network conditions in the actual application scenario, avoiding the problem of deviations caused by the disconnection between the test environment and the actual application scenario affecting the evaluation accuracy of the model training efficiency, and improving the evaluation accuracy of the model training efficiency in distributed training.

[0073] In one embodiment, steps 10421 to 10423 are described as follows:

[0074] Step 10421 , performing path cost evaluation between nodes based on the network delay between each node in the optimal intermediate node sequence to obtain a path cost index.

[0075] Optionally, the test system analyzes the network delay between adjacent nodes in the optimal intermediate node sequence, evaluates the cost of the path from the starting second target node to the ending second target node, and calculates the path cost index, i.e., the comprehensive network delay L, using the network delay formula. The longer the network delay, the longer the data transmission waiting time, and the higher the path cost. For the optimal intermediate node sequence {n1,n2,...,n k}, the network delay between adjacent nodes is l(n i ,n i+1 ),i=1,2,...,k-1, path cost index C delay The calculation formula is:

[0076]

[0077] In one embodiment, the optimal intermediate node sequence is D→B→C→E, the network delay between node D and node B is 3ms, the network delay between node B and node C is 2ms, and the network delay between node C and node E is 4ms. Therefore, the path cost index C of the optimal intermediate node sequence D→B→C→E is delay =3 / 4+2 / 4+4 / 4=2.25.

[0078] Step 10422 , based on the network packet loss rate between each node in the optimal intermediate node sequence, the path reliability between the nodes is evaluated to obtain a path reliability index.

[0079] Furthermore, the test system analyzes the network packet loss rate between adjacent nodes in the optimal intermediate node sequence to evaluate the reliability of the path. The path reliability index, namely the comprehensive network packet loss rate P, is calculated using the network packet loss rate formula. The higher the packet loss rate, the greater the possibility of data loss during data transmission, and the lower the path reliability. For the optimal intermediate node sequence {n1,n2,...,n k}, the network packet loss rate between adjacent nodes is p(n i ,n i+1 ),i=1,2,...,k-1, path reliability index R loss The calculation formula is:

[0080]

[0081] In one embodiment, the network packet loss rate between node D and node B is 0.2%, the network packet loss rate between node B and node C is 0.1%, and the network packet loss rate between node C and node E is 0.3%. Then the path reliability index R loss =(1-0.002)*(1-0.001)*(1-0.004)≈0.094.

[0082] Step 10423: Determine the comprehensive network environment parameters of the second target node based on the path cost index, the path reliability index, and the adjusted edge weights between the nodes in the optimal intermediate node sequence.

[0083] Furthermore, the test system adjusts the weight w(n) between the nodes in the optimal intermediate node sequence. i ,n i+1 ) and the sum of them determines the comprehensive edge weight. The specific formula is as follows:

[0084]

[0085] Furthermore, the test system integrates the path cost index, path reliability index and comprehensive edge weight to obtain a triplet of comprehensive network environment parameters.

[0086] Continuing with the above embodiment, the path cost indicator C delay =2.25, path reliability index R loss = = 0.094, the edge weight between node D and node B is 4, the edge weight between node B and node C is 1, and the edge weight between node C and node E is 5, then the comprehensive edge weight W = 4 + 1 + 5 = 10, therefore, the comprehensive network environment parameters of the two second target nodes D and E are (2.25, 0.994, 10).

[0087] The embodiment of the present invention determines a comprehensive and quantified integrated network environment parameter, covering network delay cost, reliability and edge weight, providing a richer and more accurate network condition basis for the subsequent determination of the transmission rate, making the description of the network environment more detailed and comprehensive, so that the multi-machine and multi-card environment that can be constructed is more in line with the complex network conditions in actual application scenarios.

[0088] In one embodiment, steps 301 to 304 are described as follows:

[0089] Step 301 : determining a first relative entropy between model structure features based on the number of model layers and the number of model parameters, and determining a second relative entropy between data scale features based on the number of samples and the feature dimension.

[0090] Optionally, the test system calculates the first relative entropy between the model structure features by analyzing the number of model layers and the number of model parameters of the target model using the relative entropy formula, wherein the first relative entropy is used to measure the degree of difference in the model structure features. The first relative entropy H M The calculation formula is as follows:

[0091] H M =L m *log(L m / P m )+P m log(P m / L m ).

[0092] Among them, L m Indicates the number of model layers, P m Indicates the number of model parameters.

[0093] Furthermore, the test system calculates the second relative entropy between data scale features by analyzing the number of samples and feature dimensions of the target data set using the relative entropy formula. The second relative entropy measures the degree of difference in data scale features. The second relative entropy H D The calculation formula is as follows:

[0094] H D =N n*log(N n / D n )+D n log(D n / N n ).

[0095] Among them, N n represents the number of samples, D n Represents the feature dimension.

[0096] Step 302 : performing correlation calculations on the attribute vectors of each parallel strategy in the parallel strategy library based on the first relative entropy and the second relative entropy, respectively, to obtain a first cross-correlation value and a second cross-correlation value of each parallel strategy in the parallel strategy library.

[0097] Optionally, the attribute vector of each parallel strategy contains characteristic information of the model structure and data scale adapted to the parallel strategy, wherein the attribute vector of each parallel strategy can be expressed as {v L ,v P ,v N ,v D}, where v L ,v P Represents the attribute values related to the number of model layers and parameters, v N ,v D Respectively represent the attribute values related to the number of samples and feature dimensions. Therefore, the test system performs an association calculation based on the first relative entropy and the attribute values related to the number of model layers and the number of parameters of each parallel strategy in the parallel strategy library, and obtains the first cross-correlation value D of each parallel strategy in the parallel strategy library. KL1 , the specific formula is as follows:

[0098]

[0099] Furthermore, the test system performs an association calculation based on the second relative entropy and the attribute values of each parallel strategy in the parallel strategy library related to the number of samples and the feature dimension to obtain a second cross-correlation value of each parallel strategy in the parallel strategy library. The specific formula is as follows:

[0100]

[0101] Step 303 : performing a difference calculation based on the first cross-correlation value and the second cross-correlation value of each parallel strategy to obtain a correlation degree difference value of each parallel strategy.

[0102] Furthermore, the test system performs a difference calculation based on the first cross-correlation value and the second cross-correlation value of each parallel strategy to obtain a correlation degree difference value ΔD of each parallel strategy. The specific formula is as follows:

[0103] ΔD=|DKL1 -D KL2 |.

[0104] Step 304 : Screen the strategies in the parallel strategy library based on the correlation degree difference value of each parallel strategy to obtain a target parallel strategy combination.

[0105] Furthermore, the test system screens the strategies in the parallel strategy library according to the correlation degree difference value of each parallel strategy to obtain the target parallel strategy combination, as specifically described in steps 3041 to 3044 .

[0106] The embodiment of the present invention matches the corresponding parallel strategy combination in the parallel strategy library according to the model structure characteristics of the target model and the data scale characteristics of the target data set. Therefore, the parallel strategy can be dynamically selected for different models and data sets, which solves the problem of the evaluation accuracy of the model training efficiency caused by avoiding a single fixed parallel strategy, and improves the evaluation accuracy of the model training efficiency in distributed training.

[0107] In one embodiment, steps 3041 to 3044 are described as follows:

[0108] Step 3041 , sorting each parallel strategy in the parallel strategy library in ascending order of correlation degree difference value to obtain sorted parallel strategies.

[0109] Optionally, the test system sorts each parallel strategy in the parallel strategy library according to the correlation difference value of each parallel strategy, obtaining sorted parallel strategies. In one embodiment, the parallel strategy library contains parallel strategies X1, X2, X3, X4, and X5, with correlation difference values of 0.015, 0.008, 0.02, 0.01, and 0.005, respectively. The test system sorts these parallel strategies in ascending order of correlation difference value, obtaining the following order of parallel strategies: X5, X2, X4, X1, X3.

[0110] Step 3042 : performing a trend change analysis based on the correlation degree difference value between two adjacent parallel strategies in the sorted parallel strategies to obtain the correlation trend change rate between the two adjacent parallel strategies.

[0111] Furthermore, the test system performs trend change analysis based on the correlation degree difference between the two adjacent parallel strategies in the sorted parallel strategies, and obtains the correlation trend change rate between the two adjacent parallel strategies. i and s i+1 , and the corresponding correlation difference value is ΔD i and ΔD i+1 , therefore, two adjacent parallel strategies s i and s i+1The correlation trend change rate T i,i+1 The calculation formula is as follows:

[0112] T i,i+1 =|ΔD i -ΔD i+1 | / [(ΔD i +ΔD i+1 ) / 2].

[0113] Step 3043 , obtaining a policy screening interval in the sorted parallel policies; the correlation trend change rates between two adjacent parallel policies in the policy screening interval are both less than or equal to a preset change rate threshold.

[0114] Furthermore, the testing system performs screening based on the correlation trend change rate between adjacent parallel strategies, combined with a preset change rate threshold, where the preset change rate threshold is set based on actual conditions. When the correlation trend change rate between two adjacent parallel strategies is less than or equal to the preset change rate threshold, these two parallel strategies and all parallel strategies between them constitute a strategy screening interval. The parallel strategies within this strategy screening interval are relatively stable in terms of the degree of correlation between model structure and data scale, and are more likely to contain parallel strategies suitable for the target model and target dataset.

[0115] In one embodiment, the preset change rate threshold is 0.25. There are parallel strategies X5, X2, X4, X1, X3, and X6 in the parallel strategy library. The correlation trend change rate between parallel strategy X5 and parallel strategy X2 is 0.05, the correlation trend change rate between parallel strategy X2 and parallel strategy X4 is 0.1, the correlation trend change rate between parallel strategy X4 and parallel strategy X1 is 0.12, the correlation trend change rate between parallel strategy X1 and parallel strategy X3 is 0.25, and the correlation trend change rate between parallel strategy X3 and parallel strategy X6 is 0.28. Therefore, the interval from parallel strategy X5 to parallel strategy X3 is determined as the strategy screening interval.

[0116] Step 3044: Combine the target parallel strategies in the strategy screening interval to obtain a target parallel strategy combination.

[0117] Furthermore, the test system combines the target parallel strategies in the strategy screening interval to obtain a target parallel strategy combination. Continuing with the above embodiment, for the strategy screening interval parallel strategies X5 to X3, the target parallel strategy combination is [X5, X2, X4, X1, X3].

[0118] The embodiment of the present invention matches the corresponding parallel strategy combination in the parallel strategy library according to the correlation degree difference value and the correlation trend change rate. Therefore, the parallel strategy can be dynamically selected for different models and data sets, which solves the problem of the evaluation accuracy of the model training efficiency caused by avoiding a single fixed parallel strategy, and improves the evaluation accuracy of the model training efficiency in distributed training.

[0119] In one embodiment, steps 401 to 404 are described as follows:

[0120] Step 401: Divide the distributed training process of the target model into multiple training phases based on the training task, and determine the number of training rounds in each training phase according to the total number of training rounds of the target model.

[0121] Optionally, the test system divides the distributed training process of the target model into multiple training phases based on the characteristics of the training task and the complexity of the target model. Each phase focuses on different training tasks. For example, the first few phases focus on rapid model convergence, while the later phases focus on improving model accuracy.

[0122] Furthermore, the test system determines the number of training rounds included in each training phase based on the total number of training rounds set for the target model and the preset rules, wherein the preset rules are: the total number of training rounds for the target model is T total , the number of training stages divided is n t . Then the number of training rounds c in each training stage is i ,i=1,2,...,n t Expressed as:

[0123]

[0124] in, Indicates rounding operation, and mod indicates remainder operation.

[0125] In one embodiment, the total number of training rounds for the standard model is T total =100, the training process is divided into n t = 5 training stages. First calculate 100mod5=0. Then the number of training rounds in each training stage is c1=c2=c3=c4=c5=20. If the total number of training rounds becomes T total =103, then 103mod5=3, c1=c2=c3=20+1=21, c4=c5=20.

[0126] Step 402 : For each training round of each training phase, in each multi-machine and multi-card environment, the target model is trained using the target parallel strategy combination and the target data set.

[0127] Furthermore, for each training round in each training stage, the test system applies the target parallel strategy combination in various multi-machine and multi-card environments, and uses the target data set to train the target model. Training is performed in multi-machine and multi-card environments and parallel strategy combinations in different network environments, which can comprehensively examine the adaptability and performance of the target model in various actual scenarios.

[0128] In one embodiment, there are three multi-machine, multi-card environments: multi-machine, multi-card environment A, multi-machine, multi-card environment B, and multi-machine, multi-card environment C. During the first round of training in the first training phase, in multi-machine, multi-card environment A, the test system distributes the target dataset to each node according to the target parallel strategy combination and initiates target model training. In multi-machine, multi-card environments B and C, the target model is trained using the same target parallel strategy combination and target dataset. This training process is repeated in each round of subsequent training phases across these three multi-machine, multi-card environments.

[0129] Step 403: Obtain the average delay time of data in the target data set propagating from the first node to the second node, the degree of fluctuation of the transmission rate of data propagating from the first node to the second node, the balance of training progress between the first node and the second node during training, and the resource utilization of computing resources in a multi-machine and multi-card environment.

[0130] Furthermore, during the training process in each multi-machine, multi-card environment, the test system measures and records the average delay time for the data in the target data set to propagate from the first node to the second node, reflecting the time cost of data transmission. At the same time, the test system monitors the degree of fluctuation in the transmission rate of data from the first node to the second node to measure the stability of network transmission. At the same time, the test system determines the balance of the training progress of the first node and the second node during the training process, and judges the consistency of the training progress of different nodes. At the same time, the test system obtains the resource utilization rate of the first node and the second node for computing resources (such as CPU, GPU, etc.) in the multi-machine, multi-card environment during the training process.

[0131] Among them, for the training progress balance of the first node and the second node during the training process, the number of training steps completed by the first node at a certain moment is step1, the number of training steps completed by the second node is step2, the total number of training steps is Step, and the training progress balance E s The calculation formula is:

[0132] E s =1-|step1-step2| / step.

[0133] Step 404 , based on the average delay time, transmission rate fluctuation, training progress balance and resource utilization of each training round in each training stage in each multi-machine and multi-card environment, obtain the indicator data of the target model in each multi-machine and multi-card environment during the training process.

[0134] Furthermore, the test system integrates the average delay time, transmission rate fluctuation, training progress balance and resource utilization of each training round in each training stage in each multi-machine and multi-card environment to obtain the indicator data of the target model in each multi-machine and multi-card environment during the training process.

[0135] The embodiments of the present invention provide a comprehensive and accurate indicator data basis for the subsequent evaluation of the impact of target parallel strategy combinations on the training efficiency of target models in different network environments, so that the accuracy of the subsequent evaluation of the training efficiency of the model in distributed training can be achieved.

[0136] Optional, see Figure 2 , Figure 2 This is a flow chart of the method for testing the efficiency of distributed model training based on multiple machines and multiple cards provided by the present invention. The execution subject of the embodiment of the present invention is the test system. Therefore, the method for testing the efficiency of distributed model training based on multiple machines and multiple cards includes:

[0137] Step 10: Perform network simulation based on the node layout and network parameters between nodes in the multi-machine multi-card system to build a multi-machine multi-card environment under different network environments.

[0138] Optionally, the multi-machine multi-card system in an embodiment of the present invention includes multiple computing nodes, each computing node is such as a CPU, GPU, etc. Therefore, the test system obtains the node layout and network parameters between each node in the multi-machine multi-card system, wherein the node layout covers information such as the physical location and topological structure of the node, and the network parameters include bandwidth, delay, packet loss rate, etc.

[0139] Furthermore, the test system uses a network simulation tool to perform network simulation based on the node layout of the multi-machine multi-card system and the network parameters between nodes, and constructs a multi-machine multi-card environment under different network conditions. In one embodiment, the multi-machine multi-card system includes 3 computing nodes, node 1 and node 2 are connected through a network with a bandwidth of 10Gbps, a delay of 1ms, and a packet loss rate of 0.1%; node 2 and node 3 are connected through a network with a bandwidth of 25Gbps, a delay of 0.5ms, and a packet loss rate of 0.05%. The test system uses a network simulation tool (such as ns-3) to construct a network topology based on these parameters. Define nodes in ns-3, set the bandwidth, delay, and packet loss rate of the links between nodes, and thus construct a multi-machine multi-card network environment that meets the above parameters.

[0140] Step 20: Match the model library and the dataset warehouse according to the test requirements to obtain the target model to be tested and the target dataset for testing.

[0141] Optionally, embodiments of the present invention pre-establish mappings between test requirements, models in a model library, and datasets in a dataset warehouse. The model library stores models of various types and structures, while the dataset warehouse stores datasets of various domains and sizes. Therefore, the test system acquires test requirements, such as testing the performance of a model on a large dataset or testing the effectiveness of a model with a specific structure on a certain type of dataset.

[0142] Furthermore, the test system matches the model library and the dataset warehouse according to the test requirements and the mapping relationship, respectively, to obtain the target model to be tested and the target dataset for testing. In one embodiment, the test requirement is to evaluate the performance of a deep learning model for image classification on a large-scale image dataset. The model library contains a variety of image classification models such as AlexNet, VGG16, and ResNet50, and the dataset warehouse contains image datasets such as CIFAR-10, CIFAR-100, and ImageNet. Based on the test requirements and the mapping relationship, the test system selects ResNet50 as the target model in the model library because of its outstanding performance in large-scale image classification tasks; and selects ImageNet as the target dataset in the dataset warehouse because of its large scale and rich image categories, which meets the test requirements for large-scale datasets.

[0143] Step 30 : Matching is performed in the parallel strategy library based on the model structure characteristics of the target model and the data scale characteristics of the target data set to obtain a target parallel strategy combination.

[0144] Optionally, the parallel strategy library in the embodiment of the present invention stores a plurality of parallel strategies targeting the model structure characteristics of different models and the data scale characteristics of data sets.

[0145] Therefore, the test system obtains the model structure characteristics of the target model (such as the number of model layers, the number of model parameters) and the data scale characteristics of the target data set (the number of data samples, the data dimension, etc.).

[0146] Furthermore, the test system matches the parallel strategy library based on the model structure characteristics of the target model and the data scale characteristics of the target data set, and finds the most suitable target parallel strategy combination by comparing the characteristics of the target model and data set with the applicable conditions of each strategy in the parallel strategy library.

[0147] Step 40 , based on a multi-machine and multi-card environment with different network environments, a target parallel strategy combination and a target data set are used to perform distributed training on the target model, and indicator data of the target model in each multi-machine and multi-card environment during the training process is obtained.

[0148] Furthermore, the test system uses a target parallel strategy combination combined with a target data set to perform distributed training on the target model in a multi-machine and multi-card environment in different network environments. During the training process, the various indicator data of the target model in each multi-machine and multi-card environment are recorded, including indicator data such as average delay time, transmission rate fluctuation, training progress balance, and computing resource utilization (CPU, GPU usage).

[0149] Step 50 , based on the indicator data in each multi-machine and multi-card environment, an evaluation is performed to obtain the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment.

[0150] Furthermore, the test system calculates the comprehensive index coefficient in each multi-machine and multi-card environment based on the average delay time, transmission rate fluctuation, training progress balance and computing resource utilization in each multi-machine and multi-card environment. The comprehensive index coefficient can be calculated by a weighted algorithm. In one embodiment, corresponding weights are assigned to each indicator data and weighted summed to obtain the comprehensive index coefficient.

[0151] Furthermore, the test system evaluates the impact of the parallel strategy combination in each multi-machine and multi-card environment on the training efficiency of the target model based on the comprehensive index coefficient in each multi-machine and multi-card environment. By comparing the comprehensive index coefficients of the parallel strategy combinations in different multi-machine and multi-card environments, conclusions are drawn as to which parallel strategy combination can enable the target model to achieve the best training efficiency in the multi-machine and multi-card environment.

[0152] The embodiment of the present invention simulates the network by analyzing the node layout and network parameters between each node in a multi-machine multi-card system, and constructs a multi-machine multi-card environment under different network environments. This simulates the network parameters under different environments in actual application scenarios, making the model test environment more consistent with the complex network conditions in actual application scenarios, and avoiding the problem of the deviation caused by the disconnection between the test environment and the actual application scenario affecting the evaluation accuracy of the model training efficiency. On the other hand, the corresponding parallel strategy combination is matched in the parallel strategy library according to the model structure characteristics of the target model and the data scale characteristics of the target data set. Therefore, the parallel strategy can be dynamically selected for different models and data sets, avoiding the problem of low resource utilization caused by a single fixed parallel strategy affecting the evaluation accuracy of the model training efficiency, thereby improving the evaluation accuracy of the model training efficiency in distributed training.

[0153] See also Figure 3 , Figure 3This is a diagram of an embodiment of an electronic device provided by an embodiment of the present invention. Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310, a processor 320, and a computer program 311 stored in the memory 310 and executable on the processor 320. When the processor 320 executes the computer program 311, the following steps are implemented:

[0154] Conduct network simulation based on the node layout and network parameters between nodes in a multi-machine and multi-card system, and build a multi-machine and multi-card environment under different network conditions;

[0155] According to the test requirements, the model library and the dataset warehouse are matched to obtain the target model to be tested and the target dataset for testing;

[0156] Based on the model structure characteristics of the target model and the data scale characteristics of the target data set, the target parallel strategy combination is obtained by matching them in the parallel strategy library;

[0157] Based on a multi-machine and multi-GPU environment with different network environments, the target model is distributedly trained using a target parallel strategy combination and a target data set, and the target model's indicator data in each multi-machine and multi-GPU environment during the training process is obtained;

[0158] Based on the indicator data in each multi-machine and multi-card environment, the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment is obtained.

[0159] See also Figure 4 , Figure 4 Detailed description of an embodiment of a computer-readable storage medium provided by an embodiment of the present invention. Figure 4 As shown, this embodiment provides a computer-readable storage medium 400 on which a computer program 311 is stored. When the computer program 311 is executed by a processor, the following steps are implemented:

[0160] Conduct network simulation based on the node layout and network parameters between nodes in a multi-machine and multi-card system, and build a multi-machine and multi-card environment under different network conditions;

[0161] According to the test requirements, the model library and the dataset warehouse are matched to obtain the target model to be tested and the target dataset for testing;

[0162] Based on the model structure characteristics of the target model and the data scale characteristics of the target data set, the target parallel strategy combination is obtained by matching them in the parallel strategy library;

[0163] Based on a multi-machine and multi-GPU environment with different network environments, the target model is distributedly trained using a target parallel strategy combination and a target data set, and the target model's indicator data in each multi-machine and multi-GPU environment during the training process is obtained;

[0164] Based on the indicator data in each multi-machine and multi-card environment, the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment is obtained.

[0165] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the multi-machine multi-card model distributed training efficiency testing method provided by the above methods. The multi-machine multi-card model distributed training efficiency testing method includes:

[0166] Conduct network simulation based on the node layout and network parameters between nodes in a multi-machine and multi-card system, and build a multi-machine and multi-card environment under different network conditions;

[0167] According to the test requirements, the model library and the dataset warehouse are matched to obtain the target model to be tested and the target dataset for testing;

[0168] Based on the model structure characteristics of the target model and the data scale characteristics of the target data set, the target parallel strategy combination is obtained by matching them in the parallel strategy library;

[0169] Based on a multi-machine and multi-GPU environment with different network environments, the target model is distributedly trained using a target parallel strategy combination and a target data set, and the target model's indicator data in each multi-machine and multi-GPU environment during the training process is obtained;

[0170] Based on the indicator data in each multi-machine and multi-card environment, the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment is obtained.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0172] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A model distributed training efficiency test system based on multiple machines and multiple cards, characterized by: It includes a test management center, a multi-machine and multi-card environment construction module, a test requirement matching module, a strategy matching module, a model training and testing module, and a training efficiency evaluation module; the test management center is connected to the multi-machine and multi-card environment construction module, the test requirement matching module, the strategy matching module, the model training and testing module, and the training efficiency evaluation module respectively to manage each module; The multi-machine and multi-card environment construction module is used to simulate the network based on the node layout and network parameters between each node in the multi-machine and multi-card system, and to build a multi-machine and multi-card environment under different network environments; The test requirement matching module is used to match the model library and the dataset warehouse according to the test requirements to obtain the target model to be tested and the target dataset for testing; A strategy matching module, configured to perform matching in a parallel strategy library based on the model structure characteristics of the target model and the data scale characteristics of the target data set to obtain a target parallel strategy combination; A model training and testing module is used to perform distributed training on the target model using the target parallel strategy combination and the target data set in a multi-machine and multi-card environment with different network environments, and obtain indicator data of the target model in each multi-machine and multi-card environment during the training process; The training efficiency evaluation module is used to evaluate the indicator data in each multi-machine and multi-card environment to obtain the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment.

2. The model distributed training efficiency testing system based on multiple machines and multiple cards according to claim 1 is characterized in that: The network simulation is performed based on the node layout and network parameters between each node in the multi-machine and multi-card system to build a multi-machine and multi-card environment under different network environments, including: Abstracting the node layout of the multi-machine multi-card system into a multi-dimensional space model; the position of each node in the multi-machine multi-card system in the multi-dimensional space model corresponds to its actual physical position; Constructing a distance matrix between nodes based on the position coordinates of each node in the multidimensional space model, and generating an initial network topology based on the distance matrix; the matrix elements of the distance matrix between two first target nodes that are connected in the initial network topology are less than or equal to a preset distance threshold; Adjusting the initial edge weight of the first target node in the initial network topology based on the network bandwidth between the first target nodes to obtain an adjusted network topology; Based on the adjusted network topology and the network delay and network packet loss rate between the first target nodes, a network simulation is performed to construct a multi-machine and multi-card environment under different network environments.

3. The model distributed training efficiency testing system based on multiple machines and multiple cards according to claim 2 is characterized in that: The performing of network simulation based on the adjusted network topology in combination with the network delay and the network packet loss rate between the first target nodes to construct a multi-machine and multi-card environment under different network environments includes: For any two second target nodes in the adjusted network topology, planning an optimal intermediate node sequence for the second target nodes in the adjusted network topology based on a shortest path algorithm combined with the adjusted edge weights of the first target nodes; Determining comprehensive network environment parameters of the second target node based on network delays, network packet loss rates, and adjusted edge weights between nodes in the optimal intermediate node sequence; determining a transmission rate of the second target node based on a comprehensive network environment parameter of the second target node, a first processing capability of the sending node, and a second processing capability of the receiving node; The network transmission process of data in the multi-machine multi-card system is simulated at the transmission rate of the second target node to construct a multi-machine multi-card environment under different network environments.

4. The model distributed training efficiency testing system based on multiple machines and multiple cards according to claim 3 is characterized in that: The determining of the comprehensive network environment parameters of the second target node based on the network delay, network packet loss rate and adjusted back-edge weight between each node in the optimal intermediate node sequence includes: Performing path cost evaluation between nodes based on the network delay between each node in the optimal intermediate node sequence to obtain a path cost index; Performing a path reliability evaluation between nodes based on the network packet loss rate between each node in the optimal intermediate node sequence to obtain a path reliability index; Based on the path cost indicator, the path reliability indicator and the adjusted edge weights between the nodes in the optimal intermediate node sequence, the comprehensive network environment parameters of the second target node are determined.

5. The model distributed training efficiency testing system based on multiple machines and multiple cards according to claim 1 is characterized in that: The multi-machine and multi-card environment based on different network environments performs distributed training on the target model using the target parallel strategy combination and the target data set, and obtains indicator data of the target model in each multi-machine and multi-card environment during the training process, including: Dividing the distributed training process of the target model into multiple training phases based on the training tasks, and determining the number of training rounds in each training phase according to the total number of training rounds of the target model; For each training round of each training phase, in each multi-machine and multi-GPU environment, the target model is trained using the target parallel strategy combination and the target dataset; Obtaining the average delay time of data in the target data set propagating from the first node to the second node, the degree of fluctuation in the transmission rate of data propagating from the first node to the second node, the balance of training progress between the first node and the second node during training, and the resource utilization of computing resources in a multi-machine and multi-card environment; Based on the average delay time, transmission rate fluctuation, training progress balance and resource utilization of each training round in each training stage in each multi-machine and multi-card environment, the indicator data of the target model in each multi-machine and multi-card environment during the training process is obtained.

6. The model distributed training efficiency testing system based on multiple machines and multiple cards according to any one of claims 1 to 5, characterized in that: The model structure characteristics include the number of model layers and the number of model parameters; the data scale characteristics include the number of samples and feature dimensions; The model structure characteristics based on the target model and the data scale characteristics of the target data set are matched in the parallel strategy library to obtain a target parallel strategy combination, including: Determine the first relative entropy between model structure features based on the number of model layers and the number of model parameters, and determine the second relative entropy between data scale features based on the number of samples and feature dimensions; Performing correlation calculation based on the first relative entropy and the second relative entropy with the attribute vector of each parallel strategy in the parallel strategy library, respectively, to obtain a first cross-correlation value and a second cross-correlation value of each parallel strategy in the parallel strategy library; Performing a difference calculation based on the first cross-correlation value and the second cross-correlation value of each parallel strategy to obtain a correlation degree difference value of each parallel strategy; Based on the correlation degree difference value of each parallel strategy, strategies are screened in the parallel strategy library to obtain the target parallel strategy combination.

7. The model distributed training efficiency testing system based on multiple machines and multiple cards according to claim 6 is characterized in that: The strategy screening is performed in the parallel strategy library based on the correlation degree difference value of each parallel strategy to obtain the target parallel strategy combination, including: Sorting each parallel strategy in the parallel strategy library in ascending order of correlation degree difference values to obtain sorted parallel strategies; Performing a trend change analysis based on the correlation degree difference value of two adjacent parallel strategies in the sorted parallel strategies to obtain the correlation trend change rate between the two adjacent parallel strategies; Obtaining a strategy screening interval in the sorted parallel strategies; wherein the correlation trend change rates between two adjacent parallel strategies in the strategy screening interval are both less than or equal to a preset change rate threshold; The target parallel strategies in the strategy screening interval are combined to obtain a target parallel strategy combination.

8. A method for testing the efficiency of distributed model training based on multiple machines and multiple graphics cards, implemented based on the system for testing the efficiency of distributed model training based on multiple machines and multiple graphics cards as claimed in any one of claims 1 to 7, characterized in that: The multi-machine and multi-GPU model distributed training efficiency testing method includes: Conduct network simulation based on the node layout and network parameters between nodes in a multi-machine and multi-card system, and build a multi-machine and multi-card environment under different network conditions; According to the test requirements, the model library and the dataset warehouse are matched to obtain the target model to be tested and the target dataset for testing; Matching the model structure characteristics of the target model and the data scale characteristics of the target data set in a parallel strategy library to obtain a target parallel strategy combination; Performing distributed training on the target model using the target parallel strategy combination and the target data set in a multi-machine and multi-card environment with different network environments, and obtaining indicator data of the target model in each multi-machine and multi-card environment during the training process; Based on the indicator data in each multi-machine and multi-card environment, the evaluation is performed to obtain the impact of the target parallel strategy combination on the training efficiency of the target model in the multi-machine and multi-card environment of each network environment.

9. An electronic device comprising: The memory and processor are characterized in that a computer software program is stored on the memory, and when the processor reads and executes the computer software program, the model distributed training efficiency testing method based on multiple machines and multiple cards as described in claim 8 is implemented.

10. A non-transitory computer-readable storage medium, characterized in that The storage medium stores a computer software program, which, when executed by the processor, implements the model distributed training efficiency testing method based on multiple machines and multiple cards as described in claim 8.

Images