Cluster network configuration method, computing device and system

Abstract

The embodiment of the invention provides a cluster network configuration method, computing equipment and a cluster network configuration system. The method comprises the following steps: receiving a first job request; the first job request is used for indicating the cluster to execute a training task; determining a target computing node from a cluster according to the resource demand of the first job request; determining whether a target network configuration parameter exists in a network configuration database based on the node information of the target computing node; wherein the network configuration database is used for storing network configuration parameters after default network configuration is optimized; under the condition that the target network configuration parameter exists in the network configuration database, issuing a second operation request to the target computing node; wherein the second operation request is determined according to the first operation request and a target network configuration parameter; the second job request is used for triggering the target computing node to execute network configuration based on the target network configuration parameter and execute the training task. According to the method, the network parameter configuration efficiency can be improved.

Description

Technical Field

[0001] This application relates to the field of distributed computing technology, and in particular, to a cluster network configuration method, a computing device, and a system. Background Art

[0002] Distributed computing is a technology that distributes computing tasks to multiple computing nodes and completes tasks through network collaboration. It can effectively handle massive data processing requests and achieve high-performance computing. To better manage a large cluster of computing nodes, a management platform can be used to receive user job requests and allocate computing node resources of the cluster to users.

[0003] However, users need to correctly configure the parameters of the computing nodes and optimize the network performance to achieve high-performance computing and exert the computing power of the computing nodes. Therefore, there is a problem of low efficiency in network parameter configuration. Summary of the Invention

[0004] Embodiments of this application provide a cluster network configuration method, a computing device, and a system to improve the efficiency of network parameter configuration.

[0005] In a first aspect, an embodiment of this application provides a cluster network configuration method, which includes:

[0006] Receiving a first job request; the first job request is used to instruct the cluster to execute a training task;

[0007] Determining a target computing node from the cluster; the target computing node is used to execute the first job request;

[0008] Based on the node information of the target computing node, determining whether the target network configuration parameters exist in the network configuration database; wherein, the network configuration database is used to store the network configuration parameters optimized from the default network configuration;

[0009] In the case where the target network configuration parameters exist in the network configuration database, sending a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0010] In the above solution, when a user requests the cluster to execute a training task, according to the resource requirements of the user request, a target computing node for executing the training task is allocated from the cluster, and based on the node information of the target computing node, the corresponding target network configuration parameters are matched in the network configuration database. Thus, when there are target network configuration parameters in the network configuration database, the network configuration of the target computing node is executed based on the target network configuration parameters. By storing multiple network configuration parameters optimized from the default network configuration in the network configuration database, after allocating a computing node, the target network configuration parameters corresponding to the node information can be efficiently matched from the network configuration database, without the need for the user to manually tune the parameters, realizing network self-adaptive optimization without the user's awareness, effectively improving the network configuration efficiency of the computing node, and being beneficial to improving the cluster performance.

[0011] In a possible implementation manner, before determining whether there are target network configuration parameters in the network configuration database based on the node information of the target computing node, the method further includes:

[0012] Generating the network configuration database based on a network performance test of the network configuration of the cluster.

[0013] In the above solution, the default network configuration of the cluster can be adjusted in advance, and the network performance after the adjustment can be tested, so as to optimize the network configuration of the cluster and retain the optimized network configuration parameters to obtain the network configuration database. Thus, after receiving the first job request, the target network configuration parameters can be quickly obtained from the network configuration database, and the network configuration can be efficiently performed according to the target network configuration parameters.

[0014] In a possible implementation manner, generating the network configuration database based on the optimization of the network configuration of the cluster includes:

[0015] Obtaining first node information, where the first node information includes the number of nodes, node parameters, node topology, and network type of the computing nodes;

[0016] According to the first node information, selecting a first test node in the cluster, testing the network performance of the test node using the first default network configuration parameters, and taking the test result of the first default network configuration parameters as the first network performance baseline corresponding to the first node information;

[0017] Testing the network performance of the first test node using the first adjusted network configuration parameters. If the test result of the first adjusted network configuration parameters is better than the first network performance baseline, adding the first adjusted network configuration parameters to the network configuration database.

[0018] In the above solution, for different node information, the default network configuration parameters and adjusted network configuration parameters can be used to test the network performance of the test nodes. The test results of the default network configuration parameters are used as the network performance baseline corresponding to the node information. When the test results of the adjusted network configuration parameters are better than the network performance baseline, the adjusted network configuration parameters are added to the network configuration database. Thus, it can be ensured that after the adjusted network configuration parameters added to the network configuration database are applied to the computing nodes, the network performance of the computing nodes is not lower than the network performance baseline. On the one hand, automatic network parameter tuning can be achieved, and automatic configuration of network parameters can be performed based on the network configuration database without manual testing by the user, greatly improving the efficiency of network parameter tuning and configuration. On the other hand, the network performance of the parameter template can be effectively guaranteed, and the reliability of network parameter tuning can be improved.

[0019] In a possible implementation, the network performance of the first test node is tested using the first adjusted network configuration parameters. If the test results of the first adjusted network configuration parameters are better than the first network performance baseline, adding the first adjusted network configuration parameters to the network configuration database includes:

[0020] The network performance of the first test node is tested using at least one set of first adjusted network configuration parameters. If there are test results of the first adjusted network configuration parameters that are better than the first network performance baseline, the second adjusted network configuration parameters are added to the network configuration database; where the second adjusted network configuration parameters are the first adjusted network configuration parameters with the best test results.

[0021] In the above solution, by adjusting the default parameters once or multiple times and using at least one set of adjusted first adjusted network configuration parameters to test the network performance of the first test node, the tuning effect of network parameters can be effectively improved, thereby enhancing the reliability of automatic configuration of network parameters.

[0022] In a possible implementation, the method further includes:

[0023] In the case where the target network configuration parameters do not exist in the network configuration database, the computing nodes are configured according to the first default network configuration parameters.

[0024] In a possible implementation, before selecting the first test node in the cluster according to the first node information, the method further includes:

[0025] Obtain the network type of the cluster and all node information;

[0026] According to the network type of the cluster and all node information, perform corresponding basic network configuration on the cluster.

[0027] In the above solution, the network environment can be automatically deployed and configured for different network types, which can effectively improve the network configuration efficiency without manual operation by the user.

[0028] In one possible implementation, according to the network type of the cluster and all node information, corresponding basic network configurations are performed on the cluster, including:

[0029] When the network type of the cluster is an IB network, the unified network management service and remote direct memory access function of the cluster are enabled;

[0030] When the network type of the cluster is a RoCE network, the traffic control function based on priority, as well as the network card IP, routing, and maximum transmission unit are configured.

[0031] In one possible implementation, the first test node is a node in the cluster that communicates with the same switch.

[0032] In the above solution, the first test node that communicates with the same switch in the cluster can be selected instead of the first test node across switches, so as to ensure better network performance of the first network performance baseline and improve the effect of network tuning.

[0033] In a second aspect, an embodiment of the present application provides a computing device, which includes a memory and a processor;

[0034] The memory and the processor are coupled;

[0035] The memory is used to store computer program instructions;

[0036] The processor is used to execute the computer program instructions so that the computing device performs: receiving a first job request; the first job request is used to instruct the cluster to execute a training task;

[0037] According to the resource requirements of the first job request, a target computing node is determined from the cluster; the target computing node is used to execute the first job request;

[0038] Based on the node information of the target computing node, it is determined whether the target network configuration parameters exist in the network configuration database; wherein, the network configuration database is used to store the network configuration parameters optimized from the default network configuration;

[0039] When the target network configuration parameters exist in the network configuration database, a second job request is sent to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0040] In a possible implementation, before the computing device determines whether the target network configuration parameter exists in the network configuration database based on the node information of the target computing node, the computing device also performs: generating the network configuration database based on a network performance test on the network configuration of the cluster.

[0041] In a possible implementation, the computing device generates a network configuration database based on optimizing the network configuration of the cluster, including:

[0042] Obtain the first node information, where the first node information includes the number of nodes, node parameters, node topology, and network type of the computing nodes;

[0043] According to the first node information, select the first test node in the cluster, test the network performance of the test node using the first default network configuration parameter, and use the test result of the first default network configuration parameter as the first network performance baseline corresponding to the first node information;

[0044] Test the network performance of the first test node using the first adjusted network configuration parameter. If the test result of the first adjusted network configuration parameter is better than the first network performance baseline, add the first adjusted network configuration parameter to the network configuration database.

[0045] In a possible implementation, when the computing device tests the network performance of the first test node using the first adjusted network configuration parameter, if the test result of the first adjusted network configuration parameter is better than the first network performance baseline, adding the first adjusted network configuration parameter to the network configuration database includes:

[0046] Test the network performance of the first test node using at least one set of first adjusted network configuration parameters. If there is a test result of the first adjusted network configuration parameter that is better than the first network performance baseline, add the second adjusted network configuration parameter to the network configuration database; where the second adjusted network configuration parameter is the first adjusted network configuration parameter with the best test result.

[0047] In a possible implementation, when there is no target network configuration parameter in the network configuration database, the computing device configures the computing node according to the first default network configuration parameter.

[0048] In a possible implementation, before the computing device selects the first test node in the cluster according to the first node information, the computing device also performs:

[0049] Obtain the network type of the cluster and all node information;

[0050] Perform the corresponding basic network configuration on the cluster according to the network type of the cluster and all node information.

[0051] In a possible implementation, the computing device executes: according to the network type of the cluster and all node information, perform corresponding basic network configuration on the cluster, including:

[0052] When the network type of the cluster is an IB network, enable the unified network management service and remote direct memory access function of the cluster;

[0053] When the network type of the cluster is a RoCE network, configure the priority-based flow control function, network card IP, routing, and maximum transmission unit.

[0054] In a possible implementation, the first test node is a node in the cluster that communicates with the same switch.

[0055] In a third aspect, an embodiment of the present application provides a distributed computing system, and the computing system includes the computing device described in the second aspect.

[0056] In a fourth aspect, an embodiment of the present application provides a management platform, including:

[0057] A scheduling module, configured to receive a first job request; the first job request is used to instruct the cluster to execute a training task;

[0058] Determine a target computing node from the cluster according to the resource requirements of the first job request; the target computing node is used to execute the first job request;

[0059] An adaptation module, configured to determine whether there are target network configuration parameters in the network configuration database based on the node information of the target computing node; wherein, the network configuration database is used to store network configuration parameters optimized from the default network configuration;

[0060] In the case where there are target network configuration parameters in the network configuration database, send a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0061] In a fifth aspect, an embodiment of the present application provides a computer-readable medium, and computer-executable instructions are stored in the computer-readable storage medium. When the computer-executable instructions are executed by a processing chip, the computing device can execute the cluster network configuration method described in the first aspect.

[0062] In a sixth aspect, an embodiment of the present application provides a computer program product, and the computer program product includes computer-executable instructions. When the computer-executable instructions are executed by a processing chip, the computing device can execute the cluster network configuration method described in the first aspect.

[0063] The cluster network configuration method, computing device, and system provided by the embodiments of the present application, when a user requests the cluster to execute a training task, allocate target computing nodes for executing the training task from the cluster according to the resource requirements of the user request, and match corresponding target network configuration parameters in the network configuration database based on the node information of the target computing nodes. Thus, when there are target network configuration parameters in the network configuration database, the network configuration of the target computing nodes can be executed based on the target network configuration parameters. By storing multiple network configuration parameters optimized from the default network configuration in the network configuration database, after allocating computing nodes, the target network configuration parameters corresponding to the node information can be efficiently matched from the network configuration database, eliminating the need for users to manually optimize parameters, realizing network self-adaptive optimization without user perception, effectively improving the network configuration efficiency of computing nodes, and being beneficial to improving the cluster performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0065] Figure 1 A schematic diagram of an application scenario provided by the embodiments of the present application;

[0066] Figure 2 A flowchart of a cluster network configuration method provided by the embodiments of the present application;

[0067] Figure 3 A flowchart of another cluster network configuration method provided by the embodiments of the present application;

[0068] Figure 4 A flowchart of yet another cluster network configuration method provided by the embodiments of the present application;

[0069] Figure 5 A schematic diagram of the structure of the management platform provided by the embodiments of the present application;

[0070] Figure 6 A schematic diagram of the structure of the computing device provided by the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0071] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0072] Distributed computing is a technology that distributes computing tasks to multiple computing nodes and collaborates through a network to complete tasks. It can break through the bottleneck of single-machine computing power, effectively handle massive data processing requests, achieve high-performance computing, and is widely used in big data volume computing scenarios such as AI large model training, climate simulation, and nuclear fusion research. To better manage a large cluster of computing nodes, a management platform can be used to receive user job requests and allocate computing node resources of the cluster to users.

[0073] However, when users use the allocated computing nodes to perform computing jobs, they need to have a certain understanding of the node information of the target computing nodes, conduct experiments on the computing nodes, correctly configure the parameters of the computing nodes, and optimize the network performance in order to achieve a high-performance network and maximize the computing power of the computing nodes. Not only is the parameter optimization difficult, but the network parameter configuration efficiency is also low.

[0074] To solve the above technical problems, the embodiments of the present application pre-test corresponding parameter templates for different node information, so that after the computing nodes are allocated, the parameter templates can be automatically matched during the operation of the user's business, realizing network self-adaptive optimization without the user's perception, eliminating the need for the user to optimize parameters, and greatly improving the network parameter configuration efficiency.

[0075] For ease of understanding, the following combines Figure 1 , and describes the application scenarios related to the embodiments of the present application.

[0076] Figure 1 is a schematic diagram of an application scenario provided by the embodiments of the present application. Please refer to Figure 1 , the cluster includes multiple computing nodes, and the management platform can manage the cluster and perform dynamic allocation of computing nodes.

[0077] Among them, the management platform pre-optimizes the default network configuration based on different node information in advance to obtain a parameter template, and stores the parameter template in the experience library. This process is only visible to the administrator of the management platform and invisible to the users who use the cluster for jobs. After receiving a job request from a user, the management platform allocates computing nodes for the user, and matches the corresponding parameter template in the experience library according to the node information of the allocated target computing node. Therefore, the management platform can directly append the parameter template to the job command line and send the computing task to the allocated computing node, so as to achieve network adaptive optimization without the user's perception, without the need for the user to adjust parameters, and greatly improve the network parameter configuration efficiency.

[0078] The embodiments of the present application do not limit the specific type of the computing node. For example, the computing node can be a server, a desktop computer, a tablet computer, or an artificial intelligence device, etc. The server can be, for example, a GPU (Graphics Processing Unit) server, a TPU (Tensor Processing Unit) server, or an AI (Artificial Intelligence) server, etc.

[0079] Next, in combination with Figure 2 , the cluster network configuration method provided by the present application will be described.

[0080] Figure 2 It is a schematic flowchart of the cluster network configuration method provided by the embodiments of the present application. The execution subject of this method can be the management platform of the cluster. As Figure 2 shown, this method may include:

[0081] S201. Receive a first job request; the first job request is used to instruct the cluster to execute a training task;

[0082] S202. Determine a target computing node from the cluster according to the resource demand of the first job request; the target computing node is used to execute the first job request;

[0083] S203. Based on the node information of the target computing node, determine whether there are target network configuration parameters in the network configuration database; wherein, the network configuration database is used to store the network configuration parameters optimized from the default network configuration;

[0084] S204. When there are target network configuration parameters in the network configuration database, send a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0085] In a specific implementation, the management platform will pre-optimize the default network configurations corresponding to different node information to obtain optimized network configuration parameters, and store the network configuration parameters in a network configuration database (experience database). When a user needs to use the computing nodes in the cluster to execute a computing task, a first job request can be sent to the management platform. The management platform can obtain the status of each computing node in the cluster and allocate at least one target computing node according to the first job request. After the management platform completes the allocation of the computing nodes, it obtains the node information of the target computing nodes allocated this time and automatically matches the target network configuration parameters corresponding to the node information in the network configuration database. The management platform can append the parameter values in the parameter template to the job command line and send the job command to the target computing nodes to start the computing task.

[0086] Exemplarily, the first job request may include at least one of the number of computing nodes, GPU type, number of GPUs, number of CPUs, memory size, and network type.

[0087] In some embodiments, before determining whether there are target network configuration parameters in the network configuration database based on the node information of the target computing nodes, the method further includes:

[0088] Generate a network configuration database based on the network configuration optimization of the cluster.

[0089] In a specific implementation, the default network configuration of the cluster can be pre-adjusted, and the network performance after the adjustment can be tested, so as to optimize the network configuration of the cluster and retain the optimized network configuration parameters to obtain a network configuration database. Thus, after receiving the first job request, the target network configuration parameters can be quickly obtained from the network configuration database, and the network configuration can be efficiently performed according to the target network configuration parameters.

[0090] The cluster network configuration method provided in this embodiment, when a user requests the cluster to execute a training task, allocates target computing nodes for executing the training task from the cluster according to the resource requirements requested by the user, and matches the corresponding target network configuration parameters in the network configuration database based on the node information of the target computing nodes. Thus, when there are target network configuration parameters in the network configuration database, the network configuration of the target computing nodes can be performed based on the target network configuration parameters. By storing multiple optimized network configuration parameters of the default network configuration in the network configuration database, after allocating the computing nodes, the target network configuration parameters corresponding to the node information can be efficiently matched from the network configuration database, without the need for the user to manually optimize the parameters, effectively improving the network configuration efficiency of the computing nodes and being beneficial to improving the cluster performance.

[0091] The embodiment of the present application also provides a cluster network configuration method. Figure 3The flowchart of another cluster network configuration method provided by an embodiment of this application. The execution subject of this method may be a management platform. As Figure 3 shown, this method includes:

[0092] S301. Obtain first node information, where the first node information includes the number of computing nodes, node parameters, node topology, and network type.

[0093] Exemplarily, the network type may include, but is not limited to, an IB network (InfiniBand Network), a RoCE (RDMA over Converged Ethernet) network, etc.

[0094] Exemplarily, the node parameters may include the number of GPUs, the number of CPUs, and / or the number of network cards. The node parameters may include GPU parameters and / or network card parameters. The node topology may include the interconnection topology of GPUs and / or the interconnection topology of network cards.

[0095] S302. According to the first node information, select a first test node in the cluster, test the network performance of the test node using first network configuration default parameters, and use the test result of the first default network configuration parameters as the first network performance baseline corresponding to the first node information.

[0096] In a specific implementation, a first test node matching the first node information may be selected in the cluster, and the default test command may be used to test the network performance of the first test node under the first network configuration default parameters to obtain the first network performance baseline.

[0097] Exemplarily, the network configuration default parameters may include GPU communication variables (such as NVIDIA Collective Communications Library, NCCL communication variables), environment variables (such as Unified Communication X, referred to as the environment variables in the UCX framework), or other variables required by users. Among them, the GPU communication variables may include, but are not limited to, communication topology algorithms, the number of queue pairs (QP) for data transmission, etc.

[0098] S303. Test the network performance of the first test node using first adjusted network configuration parameters. If the test result of the first adjusted network configuration parameters is better than the first network performance baseline, add the first adjusted network configuration parameters to the network configuration database.

[0099] In a specific implementation, the first adjusted network configuration parameters can be adjusted to obtain the first adjusted network configuration parameters. After replacing the network parameters of the first test node with the first adjusted network configuration parameters, the network performance of the first test node is tested. If the test result corresponding to the first adjusted network configuration parameters is better than the first network performance baseline, it indicates that the adjustment of the default parameters is an effective optimization. Then, the parameter values of the first adjusted network configuration parameters are added to the network configuration database, thereby completing the network optimization of the first node information.

[0100] Exemplarily, the first adjusted network configuration parameters can be stored in the network configuration database in the form of key-value pairs.

[0101] Exemplarily, the method further includes:

[0102] If the test result of the first adjusted network configuration parameters is not better than the first network performance baseline, the first adjusted network configuration parameters are not added to the network configuration database.

[0103] In a specific implementation, if the test result of the first adjusted network configuration parameters is not better than the first network performance baseline, it indicates that the adjustment effect of the first default network configuration parameters is not good. Therefore, the first adjusted network configuration parameters are discarded to ensure that in the actual network configuration, the network performance of the computing node is not lower than the first network performance baseline.

[0104] It should be noted that there are various ways to select the first test node in step S302. In a possible implementation, the first test node that communicates with the same switch in the cluster can be selected instead of the first test node across switches, so as to ensure better network performance of the first network performance baseline and improve the effect of network optimization.

[0105] S304. Receive a first job request; the first job request is used to instruct the cluster to execute a training task;

[0106] S305. Determine a target computing node from the cluster according to the resource requirements of the first job request; the target computing node is used to execute the first job request;

[0107] S306. Based on the node information of the target computing node, determine whether the target network configuration parameters exist in the network configuration database; wherein, the network configuration database is used to store the network configuration parameters optimized from the default network configuration.

[0108] S307. When there is a target network configuration parameter in the network configuration database, send a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameter; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameter and execute a training task.

[0109] It should be noted that the specific implementation manners of steps S304 - S307 can refer to S201 - 204, which will not be elaborated here.

[0110] The cluster network configuration method provided in this embodiment can, for different node information, use the default network configuration parameter and the adjusted network configuration parameter to test the network performance of the test node, take the test result of the default network configuration parameter as the network performance baseline corresponding to the node information, and when the test result of the adjusted network configuration parameter is better than the network performance baseline, add the adjusted network configuration parameter to the network configuration database, so as to ensure that after the adjusted network configuration parameter added to the network configuration database is applied to the computing node, the network performance of the computing node is not lower than the network performance baseline. On the one hand, it can realize automatic network parameter tuning and automatic configuration of network parameters based on the network configuration database, without manual testing by users, greatly improving the efficiency of network parameter tuning and configuration; on the other hand, it can effectively guarantee the network performance of the parameter template and improve the reliability of network parameter tuning.

[0111] The embodiment of the present application also provides a cluster network configuration method. Figure 4 It is a flowchart of another cluster network configuration method provided by the embodiment of the present application. The execution subject of this method can be a management platform. As Figure 4 shown, this method includes:

[0112] S401. Obtain the network type of the cluster and all node information; according to the network type of the cluster and all node information, perform corresponding basic network configuration on the cluster.

[0113] Exemplarily, the configuration of the IB network is relatively simple. When the network type of the cluster is the IB network, only the unified network management service of the cluster (such as Unified Flow Management, UFM service) and the remote direct memory access (such as remote direct memory access, RDMA) function need to be enabled to use. Optionally, the advanced features of the IB network (such as SHARP) can also be enabled.

[0114] Exemplarily, when the network type of the cluster is the RoCE network, configure the priority-based flow control function, network card IP, routing, and maximum transmission unit.

[0115] In a specific implementation, when the network type of the cluster is a RoCE network, the management platform needs to configure the priority queue number in the network card transmission packet in combination with the priority-based flow control and congestion information configured by the switch, so that the cluster can support the priority-based flow control function, ensuring that congestion of a certain type of traffic does not affect the normal forwarding of other types of traffic, and thus different types of packets on the same link do not interfere with each other. The management platform also configures the IP address and routing of the network card to ensure communication between the corresponding network cards. Additionally, the management platform can set the Maximum Transmission Unit (MTU) to the default empirical value or adjust the maximum transmission unit to improve network performance.

[0116] In this embodiment, the management platform can perform automated deployment and configuration of the network environment for different network types, effectively improving the network configuration efficiency without the need for manual operation by the user.

[0117] S402. Obtain first node information, where the first node information includes the number of computing nodes, node parameters, node topology, and network type.

[0118] S403. According to the first node information, select a first test node in the cluster, test the network performance of the test node using the first network configuration default parameters, and use the test result of the first default network configuration parameters as the first network performance baseline corresponding to the first node information.

[0119] It should be noted that the specific implementation methods of steps S402 - S403 can refer to steps S301 - S302 and will not be elaborated here.

[0120] S404. Test the network performance of the first test node using at least one set of first adjusted network configuration parameters. If there is a test result of the first adjusted network configuration parameters that is better than the first network performance baseline, add the second adjusted network configuration parameters to the network configuration database; where the second adjusted network configuration parameters are the first adjusted network configuration parameters with the best test result.

[0121] In a specific implementation, the first default network configuration parameters can be adjusted once or multiple times, and the network performance of the first test node is tested using at least one set of adjusted first adjusted network configuration parameters. If there is a test result of the first adjusted network configuration parameters that is better than the first network performance baseline, determine a set of second adjusted network configuration parameters with the best test result among at least one set of first adjusted network configuration parameters, and add the second adjusted network configuration parameters to the network configuration database.

[0122] In this embodiment, adding the second adjusted network configuration parameters with the best network performance and better than the first network performance baseline to the network configuration database can effectively improve the network parameter tuning effect and network performance.

[0123] S405. Receive a first job request; the first job request is used to instruct the cluster to execute a training task.

[0124] S406. Determine a target computing node from the cluster according to the resource requirements of the first job request; the target computing node is used to execute the first job request.

[0125] S407. Based on the node information of the target computing node, determine whether there are target network configuration parameters in the network configuration database; wherein, the network configuration database is used to store the network configuration parameters optimized from the default network configuration.

[0126] S408. When there are target network configuration parameters in the network configuration database, send a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0127] It should be noted that the specific implementation manners of steps S405 - S408 can refer to steps S201 - S204 and will not be elaborated here.

[0128] S409. When there are no target network configuration parameters in the network configuration database, configure the target computing node according to the first default network configuration parameters.

[0129] In specific implementation, if there are no target network configuration parameters in the network configuration database, it indicates that the default network configuration cannot be effectively tuned. Therefore, the target computing node is configured with the first default network configuration parameters to efficiently implement the configuration of network parameters.

[0130] The cluster network configuration method provided in this embodiment can perform network performance tests on test nodes using default network configuration parameters and adjusted network configuration parameters according to different node information. The test results of the default network configuration parameters are used as the network performance baseline corresponding to the node information. When the test results of the adjusted network configuration parameters are better than the network performance baseline, the adjusted network configuration parameters are added to the network configuration database. Thus, it can be ensured that after the adjusted network configuration parameters added to the network configuration database are applied to the computing nodes, the network performance of the computing nodes is not lower than the network performance baseline. On the one hand, it can achieve automatic network parameter tuning and automatic configuration of network parameters based on the network configuration database without manual testing by users, greatly improving the efficiency of network parameter tuning and configuration. On the other hand, it can effectively guarantee the network performance of the parameter template and improve the reliability of network parameter tuning.

[0131] An embodiment of the present application also provides a management platform. Figure 5 It is a schematic structural diagram of the management platform provided in the embodiment of the present application. As Figure 5 shown, the management platform 50 includes:

[0132] A scheduling module 51, configured to receive a first job request; the first job request is used to instruct the cluster to execute a training task;

[0133] According to the resource requirements of the first job request, determine a target computing node from the cluster; the target computing node is used to execute the first job request;

[0134] An adaptive module 52, configured to determine whether there are target network configuration parameters in the network configuration database based on the node information of the target computing node; wherein, the network configuration database is used to store network configuration parameters optimized from the default network configuration.

[0135] In the case where there are target network configuration parameters in the network configuration database, send a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameters; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameters and execute the training task.

[0136] In a possible implementation, the management platform 50 further includes:

[0137] An optimization module 53, configured to generate a network configuration database based on the network configuration optimization of the cluster.

[0138] In a possible implementation, the optimization module 53 is specifically configured to:

[0139] Obtain first node information, where the first node information includes the number of computing nodes, node parameters, node topology, and network type;

[0140] According to the first node information, select a first test node in the cluster, test the network performance of the test node using the first default network configuration parameters, and use the test result of the first default network configuration parameters as the first network performance baseline corresponding to the first node information;

[0141] Test the network performance of the first test node using the first adjusted network configuration parameters. If the test result of the first adjusted network configuration parameters is better than the first network performance baseline, add the first adjusted network configuration parameters to the network configuration database.

[0142] In a possible implementation, the optimization module 53 is further specifically configured to:

[0143] Test the network performance of the first test node using at least one set of first adjusted network configuration parameters. If there is a test result of the first adjusted network configuration parameters that is better than the first network performance baseline, add the second adjusted network configuration parameters to the network configuration database; where the second adjusted network configuration parameters are the first adjusted network configuration parameters with the best test result.

[0144] In a possible implementation, the adaptive module 52 is further configured to:

[0145] In the case where the target network configuration parameters do not exist in the network configuration database, configure the target computing node according to the first default network configuration parameters.

[0146] In a possible implementation, the management platform 50 further includes:

[0147] A configuration module 54, configured to obtain the network type of the cluster and all node information;

[0148] Perform corresponding basic network configuration on the cluster according to the network type of the cluster and all node information.

[0149] In a possible implementation, the configuration module 54 is specifically configured to:

[0150] When the network type of the cluster is an IB network, enable the unified network management service and the remote direct memory access function of the cluster;

[0151] When the network type of the cluster is a RoCE network, configure the priority-based flow control function and the network card IP, routing, and maximum transmission unit.

[0152] In a possible implementation, the first test node is a node in the cluster that communicates with the same switch.

[0153] It should be noted that the management platform provided in the embodiments of the present application can be used to execute the cluster network configuration method shown in the above method embodiments. The implementation principles and technical effects are similar, and will not be elaborated here.

[0154] The embodiments of the present application further provide a computing system, including a cluster and a management platform; wherein, the cluster includes multiple computing nodes.

[0155] The computing system provided in this embodiment can be used to execute the cluster network configuration method shown in the above method embodiments. The implementation principles and technical effects are similar, and will not be elaborated here.

[0156] Figure 5 It is a schematic structural diagram of a computing device provided in the embodiments of the present application. The computing device can be a single server or a collection of multiple servers. As Figure 5 shown, the computing device includes:

[0157] A processor 291, and the computing device further includes a memory 292; it may also include a communication interface 293 and a bus 294. Among them, the processor 291, the memory 292, and the communication interface 293 can communicate with each other through the bus 294. The communication interface 293 can be used for information transmission. The processor 291 can call the logical instructions in the memory 292 to execute the method of the above embodiments.

[0158] In addition, when the logical instructions in the above-mentioned memory 292 are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium.

[0159] The memory 292, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of the present application. The processor 291 executes functional applications and data processing by running the software programs, instructions, and modules stored in the memory 292, that is, implements the methods in the above method embodiments.

[0160] The memory 292 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 292 may include high-speed random access memory and may also include non-volatile memory.

[0161] The embodiments of the present application also provide a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processing chip, they can be used to execute the cluster network configuration method shown in the foregoing method embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0162] The embodiments of the present application also provide a computer program product, which includes computer-executable instructions. When the computer-executable instructions are executed by a processing chip, they can be used to execute the cluster network configuration method shown in the foregoing method embodiments. The implementation principles and technical effects are similar and will not be elaborated here.

[0163] All or part of the steps of implementing the foregoing method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a readable memory. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing memory (storage medium) includes: read-only memory (abbreviation: ROM), RAM, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0164] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processing unit of a general-purpose computer, a special-purpose computer, an embedded processing machine, or other programmable terminal devices to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0165] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable terminal device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured product including an instruction device, and the instruction device implements the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0166] These computer program instructions can also be loaded onto a computer or other programmable terminal device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one process or multiple processes and / or one block or multiple blocks. Figure 1 one process or multiple processes and / or Figure 1 blocks or multiple blocks.

[0167] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalent technologies, the embodiments of the present application are also intended to include these modifications and variations.

[0168] In the embodiments of the present application, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising one..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element. The term "or" and its variants may refer to "and / or". In the embodiments of the present application, terms such as "first", "second" are used to distinguish similar objects and do not necessarily have to be used to describe a specific order or sequence. In the embodiments of the present application, "a plurality of" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0169] It should be understood that the step numbers in the embodiments of the present application do not constitute a limitation on the execution order, and the execution order of the steps in each embodiment should be determined according to its internal logic.

[0170] After considering the specification and the invention disclosed in practice, those skilled in the art will easily think of other implementation solutions of the present application. The embodiments of the present application are intended to cover any variations, uses or adaptive changes of the embodiments of the present application, and these variations, uses or adaptive changes follow the general principles of the embodiments of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the embodiments of the present application.

Claims

1. A cluster network configuration method, characterized in that, The method includes: Receiving a first job request; the first job request is used to instruct the cluster to execute a training task; Determining a target computing node from the cluster; the target computing node is used to execute the first job request; Based on the node information of the target computing node, determining whether there is a target network configuration parameter in the network configuration database; wherein, the node information includes the number of nodes, node parameters, node topology, and network type; the network configuration database is used to store network configuration parameters optimized from default network configurations; the target network configuration parameter is the optimized network configuration parameter corresponding to the target computing node; When the target network configuration parameter exists in the network configuration database, sending a second job request to the target computing node; wherein, the second job request is determined according to the first job request and the target network configuration parameter; the second job request is used to trigger the target computing node to perform network configuration based on the target network configuration parameter and execute the training task.

2. The method according to claim 1, wherein Before determining whether there is the target network configuration parameter in the network configuration database based on the node information of the target computing node, the method further includes: Generating the network configuration database based on network performance testing of the network configuration of the cluster.

3. The method according to claim 2, characterized in that, The generating the network configuration database based on network performance testing of the network configuration of the cluster includes: Obtaining first node information; According to the first node information, selecting a first test node in the cluster, testing the network performance of the test node using first default network configuration parameters, and using the test result of the first default network configuration parameters as the first network performance baseline corresponding to the first node information; Testing the network performance of the first test node using first adjusted network configuration parameters, and if the test result of the first network configuration adjustment parameters is better than the first network performance baseline, adding the first adjusted network configuration parameters to the network configuration database.

4. The method according to claim 3, wherein The testing the network performance of the first test node using first adjusted network configuration parameters, and if the test result of the first adjusted network configuration parameters is better than the first network performance baseline, then adding the first adjusted network configuration parameters to the network configuration database includes: Testing the network performance of the first test node using at least one set of first adjusted network configuration parameters, and if there is a test result of the first adjusted network configuration parameters that is better than the first network performance baseline, adding second adjusted network configuration parameters to the network configuration database; wherein, the second adjusted network configuration parameters are the first adjusted network configuration parameters with the best test result.

5. The method according to claim 3 or 4, characterized in that, The method further includes: When the target network configuration parameter does not exist in the network configuration database, configuring the target computing node according to the first default network configuration parameter.

6. The method according to any one of claims 3-5, characterized in that, Before selecting the first test node in the cluster according to the first node information, the method further includes: Obtaining the network type of the cluster and all node information; Execute corresponding basic network configuration for the cluster according to the network type of the cluster and all node information.

7. The method according to claim 6, wherein The executing corresponding basic network configuration for the cluster according to the network type of the cluster and all node information includes: When the network type of the cluster is an IB network, enable the unified network management service and the remote direct memory access function of the cluster; When the network type of the cluster is a RoCE network, configure the priority-based flow control function, the network card IP, the routing, and the maximum transmission unit.

8. The method according to any one of claims 3-7, characterized in that The first test node is a node in the cluster that communicates with the same switch.

9. A computing device, characterized in that, The computing device includes a memory and a processor; The memory and the processor are coupled; The memory is used to store computer program instructions; The processor is configured to execute the computer program instructions so that the computing device executes the cluster network configuration method according to any one of claims 1-8.

10. A distributed computing system, characterized in that, The computing system includes the computing device according to claim 9.

Images