Mirror image file construction method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a mirror image file construction method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining driving source codes of M versions corresponding to a first model, and application software associated with the driving source code of each version; installing and testing the driving source codes of the M versions and the associated application software one by one to obtain a test result; based on at least one first performance index value and at least one second performance index value included in the test result, determining a target version in the M versions, and obtaining a target driving source code corresponding to the target version and associated target associated application software; according to the target driving source code, the target associated application software and a preset file, constructing a target mirror image file corresponding to the training cluster; and sending the target mirror image file to a management node of the training cluster. The problems that the deployment process of the training cluster environment is complex, a large amount of manpower cost is consumed, and abnormities are prone to occurring in the deployment process can be solved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for constructing an image file, a computer device and a storage medium. Background Art

[0002] With the development of cloud computing and big data technology, artificial intelligence models, with their huge data throughput and complex algorithm structure, have not only aroused extensive discussion in academia, but also demonstrated great potential and value in commercial applications. However, the challenges of intelligent model training are multifaceted. For example, model training requires huge computing resources. For another example, the construction and deployment process of model training clusters is complicated.

[0003] The current model training cluster deployment process is complicated. It not only requires the deployment of operating systems, processor drivers, network card drivers and other application software for each computing node, the installation of container components, cluster management components and cluster resource monitoring components, etc., which consumes a lot of manpower costs. In addition, during the deployment process, problems such as driver incompatibility or network card driver incompatibility, component version dependency issues, abnormal startup of computing node container services, and incomplete operating system configuration may occur. Summary of the invention

[0004] In view of this, the present invention provides a method, device, computer equipment and storage medium for building an image file to solve the problems that the deployment process of the training cluster environment is complicated, consumes a lot of manpower costs and is prone to abnormalities during the deployment process.

[0005] In a first aspect, the present invention provides a method for constructing an image file, the method comprising: obtaining M versions of driver source code corresponding to a first model, and application software associated with each version of the driver source code, wherein the first model is a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes are the same; installing and testing the M versions of the driver source code and the associated application software one by one to obtain test results, wherein the test results include at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code; determining a target version among the M versions based on at least one first performance indicator value and at least one second performance indicator value, and obtaining a target driver source code corresponding to the target version, and a target associated application software associated with the target driver source code; constructing a target image file corresponding to the training cluster according to the target driver source code, the target associated application software and a preset file, wherein the preset file represents an associated file of a preset operating system corresponding to the N computing nodes in the training cluster; and sending the target image file to a management node of the training cluster.

[0006] Based on the method of the first aspect above, the target driver source code corresponding to the N computing nodes in the training cluster and the target associated application software associated with the target driver source code can be obtained, that is, the processor driver source code and the associated application software required for each computing node in the training cluster can be determined. In addition, the target image file can be constructed according to the associated files of the target driver source code, the target associated application software and the preset operating system, so that the target image file integrates the processor driver source code and the associated application software required for each computing node in the training cluster, and the associated various files of the preset operating system, that is, the compatibility problem of the processor driver source code and the application software, and the optimal selection problem of the associated various files of the preset operating system are pre-processed. It is only necessary to send the target image file to the management node of the training cluster, so that the management node can send the target image file to each computing node for training cluster environment deployment, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, improving the deployment efficiency, and avoiding the incompatibility of the driver source code and the abnormal problems of the application software dependency.

[0007] In an optional implementation, M versions of driver source code and associated application software are installed and tested one by one to obtain test results, including: installing a preset operating system and configuring a software package library of the preset operating system, the software package library including multiple software packages; calling a first software package that each version of the driver source code depends on and a second software package that the associated application software depends on from the software package library one by one, compiling and installing each version of the driver source code and the associated application software on the preset operating system, and testing at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code through a preset test tool.

[0008] In an embodiment of the present invention, the performance indicators of each version of the driver source code and the associated application software can be tested on the installed preset operating system, and the first software package that each version of the driver source code depends on and the second software package that the associated application software depends on can be obtained, so as to facilitate the subsequent determination of the driver source code with the best performance based on at least one first performance indicator value and at least one second performance indicator value.

[0009] In an optional embodiment, at least one first performance indicator value includes the throughput, delay duration and error rate of processing requests corresponding to each version of the driver source code, and at least one second performance indicator value includes the average processor usage, average memory usage and average disk usage corresponding to the associated application software; based on the at least one first performance indicator value and the at least one second performance indicator value, determining the target version among the M versions includes: obtaining a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage, a fifth weight corresponding to the average memory usage and a sixth weight corresponding to the average disk usage; obtaining a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software.

[0010] Based on a preset algorithm, at least one first performance indicator value and at least one second performance indicator value, a score value corresponding to each version of the driver source code is calculated; and the version with the highest score value among the M versions is determined as the target version;

[0011] The preset algorithm is: Q = μ*(α1*A+β1*B+γ1*C)+(1-μ)*(α2*D+β2*E+γ2*F). Where μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents throughput; B represents latency; C represents error rate; D represents average processor usage; E represents average memory usage; and F represents average disk usage.

[0012] In an implementation manner of the present invention, a score value corresponding to each version of the driver source code can be calculated based on a preset algorithm, at least one first performance indicator value and at least one second performance indicator value, and then a target driver source code corresponding to the target version can be obtained, thereby selecting the driver source code with the most suitable performance.

[0013] In an optional implementation, the preset files include the target network card driver source code, the third software package on which the target network card driver source code depends, the first software package, the second software package, the operating system files corresponding to the preset operating system, the cluster management related files of the training cluster, and the resource monitoring related files.

[0014] In the implementation manner of the present invention, the target image file can be pre-integrated with the target network card driver source code, the third software package on which the target network card driver source code depends, the first software package, the second software package, the operating system files corresponding to the preset operating system, the cluster management related files of the training cluster and the resource monitoring related files, so as to pre-process the compatibility issues of the network card driver source code, the installation dependency issues of the network card driver source code and the target driver source code, the operating system and the optimal selection issues of various components, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, and improving the deployment efficiency.

[0015] In an optional embodiment, a target image file corresponding to a training cluster is constructed according to a target driver source code, a target associated application software and a preset file, including: obtaining a first configuration file, the first configuration file being used to indicate a construction order among the target driver source code, the target associated application software and all files included in the preset file; based on the first configuration file, in the order of operating system files, a first software package, a second software package, a third software package, target driver source code, target associated application software, target network card driver source code, cluster management related files and resource monitoring related files, sequentially constructing the target image file corresponding to the training cluster.

[0016] In the implementation manner of the present invention, the target image file can be constructed according to the construction order indicated by the first configuration file, so that the target image file can be directly deployed to the management node and each computing node in the training cluster.

[0017] In an optional embodiment, the method also includes: obtaining P versions of network card driver source code corresponding to the second model, the second model is the infinite bandwidth network card model corresponding to N computing nodes in the training cluster, and the infinite bandwidth network card models corresponding to the N nodes are the same; installing and testing the P versions of the network card driver source code one by one to obtain the network card test results, the network card test results include at least one third performance indicator value corresponding to each version of the network card driver source code; based on the at least one third performance indicator value, determining a first version in the P versions, and obtaining the target network card driver source code corresponding to the first version.

[0018] In the implementation manner of the present invention, the target network card driver source code corresponding to N computing nodes in the training cluster can be obtained, that is, the infinite bandwidth network card driver source code required by each computing node in the training cluster can be determined.

[0019] In an optional implementation, P versions of network card driver source code are installed and tested one by one to obtain a network card test result, including: calling a third software package that each version of the network card driver source code depends on from a software package library one by one, compiling and installing each version of the network card driver source code on a preset operating system, and testing at least one third performance indicator value corresponding to each version of the network card driver source code through a network card testing tool.

[0020] In an embodiment of the present invention, the performance indicators of each version of the network card driver source code can be tested on the installed preset operating system, and the third software package that each version of the network card driver source code depends on can be obtained, so as to facilitate the subsequent determination of the network card driver source code with the best performance based on at least one third performance indicator value.

[0021] In a second aspect, the present invention provides an image file construction device, which includes: a transceiver module, used to obtain M versions of driver source code corresponding to a first model, and application software associated with each version of the driver source code, the first model is a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes are the same; a processing module, used to install and test the M versions of the driver source code and the associated application software one by one to obtain test results, the test results including at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code.

[0022] The processing module is also used to determine the target version among M versions based on at least one first performance indicator value and at least one second performance indicator value, and obtain the target driver source code corresponding to the target version, and the target associated application software associated with the target driver source code; the processing module is also used to construct the target image file corresponding to the training cluster according to the target driver source code, the target associated application software and the preset file, and the preset file represents the associated file of the preset operating system corresponding to the N computing nodes in the training cluster; the transceiver module is also used to send the target image file to the management node of the training cluster.

[0023] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image file construction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0024] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image file construction method of the first aspect or any corresponding embodiment thereof.

[0025] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the image file construction method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 is a schematic diagram of a training cluster deployment process according to a related technology of an embodiment of the present invention;

[0028] Figure 2 is a topology diagram of an image file construction system according to an embodiment of the present invention;

[0029] Figure 3 is a flow chart of a method for constructing an image file according to an embodiment of the present invention;

[0030] Figure 4 is a schematic diagram of a target image file installation process according to an embodiment of the present invention;

[0031] Figure 5 is a structural block diagram of an image file construction device according to an embodiment of the present invention;

[0032] Figure 6 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] The embodiment of the present invention is applied to the scenario of training cluster environment deployment. If any computing node has a problem in the training cluster environment deployment, it will greatly delay the deployment progress and affect the model training effect. As described in the background technology, the current model training cluster deployment process is complicated. Figure 1 As shown, Figure 1 is a schematic diagram of a training cluster deployment process according to a related technology of an embodiment of the present invention; Figure 1 In the process, the operating system is deployed through the Preboot Execution Environment (PXE) technology; the driver is deployed and the compatibility of the driver is determined. If not, the driver is reselected and a suitable driver is selected; if compatible, the application software is deployed and the software dependency is determined. If so, the application software is reselected and a suitable application software is selected; if not, the cluster management component is deployed. After all software is installed, the cluster environment deployment is completed.

[0035] Figure 1 There are many problems with the training cluster deployment method shown. For example, due to the lack of experience in driver selection, it is difficult to choose the most suitable driver to use with the hardware, and various compatibility issues need to be resolved during the adaptation process; how to choose the upper-layer application version, which version can have better compatibility with the driver and hardware, and have better performance issues; what software is used to monitor various parameters when the training cluster is used; how to manage the cluster and how to ensure the stability of the cluster; how to ensure the network stability of each computing node in the training cluster; how to deal with the interaction between each module and the operating system itself, etc.

[0036] In order to solve the above problems, an embodiment of the present invention provides a method for constructing an image file, which constructs a target image file corresponding to various files required for the deployment of a training cluster environment, so as to simplify the deployment process of the training cluster environment, reduce the deployment cost of the training cluster environment deployment, and improve the deployment efficiency.

[0037] Below Figure 2 Taking the image file construction system 200 as an example, the method provided in the embodiment of the present application is described. Figure 2 It is only a schematic diagram and does not constitute a limitation on the applicable scenarios of the technical solution provided in this application.

[0038] like Figure 2 As shown, Figure 2 It is a topological diagram of the image file construction system according to an embodiment of the present invention. Figure 2 In the embodiment, the image file construction system 200 may include an image file construction device 201 , a management node 202 , a first computing node 203 and a second computing node 204 .

[0039] In the embodiment of the present invention, the image file construction device 201 can be any device with communication capability and computing capability. For example, the image file construction device can be a server, a cloud server or a virtual machine.

[0040] In the embodiment of the present invention, the management node 202, the first computing node 203 or the second computing node 204 can be any device with communication capability and computing capability. The management node 202, the first computing node 203 and the second computing node 204 are nodes in the training cluster. The management node 202 is used to manage and monitor the first computing node 203 and the second computing node 204.

[0041] Figure 2 The image file construction system 200 shown is only used as an example and is not used to limit the technical solution of the present application. Those skilled in the art should understand that in the specific implementation process, the image file construction system 200 can also include other devices, and the number of computing nodes can also be determined according to specific needs without limitation.

[0042] According to an embodiment of the present invention, an embodiment of a method for building an image file is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] In this embodiment, a method for constructing an image file is provided, which can be used in the above-mentioned image file construction device. Figure 3 is a flow chart of a method for constructing an image file according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0044] S301: Obtain M versions of driver source codes corresponding to a first model, and application software associated with each version of the driver source code.

[0045] The first model is the processor model corresponding to the N computing nodes in the training cluster, and the processor models corresponding to the N nodes are the same. Optionally, the processor may be a graphics processing unit (GPU).

[0046] It is understandable that a processor of one model may correspond to multiple versions of driver source codes. Different driver source codes are associated with different application software. The associated application software may be one or more application software.

[0047] S302: Install and test the M versions of driver source codes and associated application software one by one to obtain test results.

[0048] The test result includes at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code.

[0049] In some optional embodiments, the image file building device installs a preset operating system and configures a software package library of the preset operating system; calls the first software package that each version of the driver source code depends on, and the second software package that the associated application software depends on, from the software package library one by one, compiles and installs each version of the driver source code and the associated application software on the preset operating system, and tests at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code through a preset testing tool.

[0050] The software package library includes multiple software packages. The software package library can be a yum source. The software package library refers to a warehouse for storing and managing the software package manager (Red Hat Package Manager, RPM).

[0051] The preset operating system may be a pure version operating system.

[0052] In the embodiment of the present invention, at least one first performance indicator value includes throughput, delay duration, and error rate of request processing corresponding to each version of the driver source code.

[0053] In the embodiment of the present invention, the at least one second performance indicator value includes an average processor usage, an average memory usage, and an average disk usage corresponding to the associated application software.

[0054] In an embodiment of the present invention, the associated application software may be a deep neural network library (CUDA Deep Neural Network library, cuDNN), a container (docker), a collective communication library (NVIDIA Collective Communications Library, NCCL), and a network manager (nvidia-fabric-manager).

[0055] In the embodiment of the present invention, the preset prediction tool may be a driver test tool and an application software test tool. For example, the driver test tool may be a driver verifier (Driver Verifier). The application software test tool may be a load runner (LoadRunner) or a debugger (GNU Debugger).

[0056] It can be understood that, in the process of installing and testing the M versions of driver source codes and associated application software one by one, the image file construction device can call multiple software packages included in the software package library.

[0057] S303: Determine a target version from the M versions based on at least one first performance indicator value and at least one second performance indicator value, and obtain a target driver source code corresponding to the target version and a target-associated application software associated with the target driver source code.

[0058] In some optional embodiments, the image file construction device obtains a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage, a fifth weight corresponding to the average memory usage, and a sixth weight corresponding to the average disk usage; obtains a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software; calculates a score value corresponding to each version of the driver source code based on a preset algorithm, at least one first performance indicator value, and at least one second performance indicator value; and determines the version with the highest score value among the M versions as the target version.

[0059] Among them, the preset algorithm is: Q=μ*(α1*A+β1*B+γ1*C)+(1-μ)*(α2*D+β2*E+γ2*F).

[0060] Among them, μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents throughput; B represents delay time; C represents error rate; D represents average processor utilization; E represents average memory utilization; F represents average disk utilization.

[0061] It is understandable that the performance of the driver source code of the processor is not only related to the performance of the driver source code itself, but also to the performance of the application associated with the driver source code. Therefore, the score value corresponding to each version of the driver source code can be determined based on the first coefficient corresponding to each version of the driver source code and the second coefficient corresponding to the associated application software.

[0062] S304: Construct a target image file corresponding to the training cluster according to the target driver source code, target associated application software and preset files.

[0063] The preset files represent the associated files of the preset operating system corresponding to the N computing nodes in the training cluster. The preset files include the target network card driver source code, the third software package that the target network card driver source code depends on, the first software package, the second software package, the operating system files corresponding to the preset operating system, the cluster management related files of the training cluster, and the resource monitoring related files.

[0064] Among them, the cluster management related files and resource monitoring related files can be Data Center GPU Manager (DCGM), Parallel Distributed Shell (PDSh), and open Lightweight Directory Access Protocol (openldap).

[0065] In some optional embodiments, the image file construction device obtains a first configuration file; based on the first configuration file, in the order of operating system files, first software package, second software package, third software package, target driver source code, target associated application software, target network card driver source code, cluster management related files and resource monitoring related files, the target image file corresponding to the training cluster is constructed in sequence.

[0066] The first configuration file is used to indicate the construction order among the target driver source code, the target associated application software and all files included in the preset file.

[0067] It is understandable that there is a specific installation order between the operating system files, the first software package, the second software package, the third software package, the target driver source code, the target associated application software, the target network card driver source code, the cluster management related files and the resource monitoring related files. If the target image file is not built in the installation order, the deployment of the training cluster will fail or anomalies will occur. For example, before installing DCGM, you need to install the network manager (nvidia-fabric-manager) and the compiler collection (GNU Compiler Collection, GCC) required to compile the driver.

[0068] In some optional embodiments, the image file building device obtains P versions of network card driver source code corresponding to the second model; installs and tests the P versions of the network card driver source code one by one to obtain a network card test result, and the network card test result includes at least one third performance indicator value corresponding to each version of the network card driver source code; based on the at least one third performance indicator value, determines a first version among the P versions, and obtains a target network card driver source code corresponding to the first version.

[0069] The second model is the InfiniBand network card model corresponding to the N computing nodes in the training cluster, and the InfiniBand network card models corresponding to the N nodes are all the same. The InfiniBand network card is also called an InfiniBand Card (IB) card.

[0070] In one example, an image file construction device calls the third software package that each version of the network card driver source code depends on from the software package library one by one, compiles and installs each version of the network card driver source code on a preset operating system, and tests at least one third performance indicator value corresponding to each version of the network card driver source code through a network card testing tool.

[0071] Among them, at least one third performance indicator includes the network card throughput, network card delay duration, number of concurrent connections and connection establishment duration corresponding to each version of the network card driver source code.

[0072] In the embodiment of the present invention, the network card testing tool may be a custom script, a load testing tool, or the like.

[0073] In one example, the image file construction device obtains the seventh weight corresponding to the network card throughput, the eighth weight corresponding to the network card delay duration, the ninth weight corresponding to the number of concurrent connections, and the tenth weight corresponding to the connection establishment duration; based on the first algorithm, the network card throughput, the network card delay duration, the number of concurrent connections, and the connection establishment duration, the network card score value corresponding to each version of the network card driver source code is calculated, and the version with the highest network card score value among P versions is determined as the first version.

[0074] The first algorithm is: S=α3*H+β3*I+γ3*J+θ*K.

[0075] Among them, α3 represents the seventh weight; β3 represents the eighth weight; γ3 represents the ninth weight; θ represents the tenth weight; H represents the network card throughput; I represents the network card delay duration; J represents the number of concurrent connections; and K represents the connection establishment time.

[0076] It is understandable that since the network card throughput, network card delay time, number of concurrent connections and connection establishment time have different impacts on the performance of the network card driver source code, the network card score value corresponding to each version of the network card driver source code can be accurately determined based on the first algorithm, and then the best target network card driver source code can be selected.

[0077] S305: Send the target image file to the management node of the training cluster.

[0078] The target image file may carry parameter configuration information.

[0079] Exemplarily, the image file construction device sends the target image file to the management node of the training cluster. The management node receives the target image file and sends the target image file to each computing node in the training cluster.

[0080] like Figure 4 As shown, Figure 4 Schematic diagram of the target image file installation process according to an embodiment of the present invention; the management node and each computing node can execute:

[0081] S401: Install the target image file through the PXE technology.

[0082] S402: Monitor the installation progress of the target image file through a monitoring tool on the management node.

[0083] S403: Based on the installation progress, determine whether the target image file has been installed; if not, continue the installation.

[0084] S404: If the installation is complete, restart and configure each module in the target image file based on the parameter configuration information, and the training cluster deployment is completed.

[0085] based on Figure 3 According to the method, the image file construction device can obtain the target driver source code corresponding to N computing nodes in the training cluster, and the target associated application software associated with the target driver source code, so as to determine the processor driver source code and the associated application software required for each computing node in the training cluster. In addition, the target image file can be constructed according to the associated files of the target driver source code, the target associated application software and the preset operating system, so that the target image file integrates the processor driver source code and the associated application software required for each computing node in the training cluster, and the various associated files of the preset operating system, that is, the compatibility problem of the processor driver source code and the application software, and the optimal selection problem of the various associated files of the preset operating system are pre-processed. It is only necessary to send the target image file to the management node of the training cluster, so that the management node can send the target image file to each computing node for training cluster environment deployment, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, improving the deployment efficiency, and avoiding the abnormal problems of driver source code incompatibility and application software dependency.

[0086] In this embodiment, a mirror file construction device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0087] This embodiment provides a mirror file construction device, such as Figure 5 As shown, Figure 5 is a structural block diagram of a device for constructing an image file according to an embodiment of the present invention; the device comprises:

[0088] The transceiver module 501 is used to obtain M versions of driver source code corresponding to the first model, and the application software associated with each version of the driver source code. The first model is the processor model corresponding to N computing nodes in the training cluster, and the processor models corresponding to the N nodes are the same.

[0089] Processing module 502 is used to install and test M versions of driver source code and associated application software one by one to obtain test results, which include at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code.

[0090] The processing module 502 is further used to determine a target version among the M versions based on at least one first performance indicator value and at least one second performance indicator value, and obtain a target driver source code corresponding to the target version, and a target-associated application software associated with the target driver source code.

[0091] The processing module 502 is also used to construct a target image file corresponding to the training cluster according to the target driver source code, the target associated application software and the preset file, where the preset file represents the associated file of the preset operating system corresponding to the N computing nodes in the training cluster.

[0092] The transceiver module 502 is also used to send the target image file to the management node of the training cluster.

[0093] In some optional implementations, the processing module 502 is specifically used to install a preset operating system and configure a software package library of the preset operating system, where the software package library includes multiple software packages; call the first software package that each version of the driver source code depends on, and the second software package that the associated application software depends on, from the software package library one by one, compile and install each version of the driver source code and the associated application software on the preset operating system, and test at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code through a preset testing tool.

[0094] In some optional embodiments, at least one first performance indicator value includes the throughput, latency, and error rate of processing requests corresponding to each version of the driver source code, and at least one second performance indicator value includes the average processor usage, average memory usage, and average disk usage corresponding to the associated application software.

[0095] The processing module 502 is also specifically used to obtain a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage, a fifth weight corresponding to the average memory usage, and a sixth weight corresponding to the average disk usage; obtain a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software; calculate a score value corresponding to each version of the driver source code based on a preset algorithm, at least one first performance indicator value, and at least one second performance indicator value; and determine the version with the highest score value among the M versions as the target version.

[0096] The default algorithm is:

[0097] Q=μ*(α1*A+β1*B+γ1*C)+(1-μ)*(α2*D+β2*E+γ2*F)

[0098] Among them, μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents throughput; B represents delay time; C represents error rate; D represents average processor utilization; E represents average memory utilization; F represents average disk utilization.

[0099] In some optional implementations, the preset files include the target network card driver source code, the third software package on which the target network card driver source code depends, the first software package, the second software package, the operating system files corresponding to the preset operating system, cluster management related files and resource monitoring related files.

[0100] In some optional embodiments, the processing module 502 is also specifically used to obtain a first configuration file, which is used to indicate the construction order between the target driver source code, the target associated application software and all files included in the preset file; based on the first configuration file, the target image file corresponding to the training cluster is constructed in sequence according to the order of operating system files, first software package, second software package, third software package, target driver source code, target associated application software, target network card driver source code, cluster management related files and resource monitoring related files.

[0101] In some optional embodiments, the transceiver module 501 is also used to obtain P versions of network card driver source code corresponding to the second model, where the second model is the infinite bandwidth network card model corresponding to N computing nodes in the training cluster, and the infinite bandwidth network card models corresponding to the N nodes are the same; the processing module 502 is also used to install and test the P versions of the network card driver source code one by one to obtain the network card test results, and the network card test results include at least one third performance indicator value corresponding to each version of the network card driver source code; the processing module 502 is also used to determine the first version in the P versions based on the at least one third performance indicator value, and obtain the target network card driver source code corresponding to the first version.

[0102] In some optional implementations, the processing module 502 is further specifically used to call the third software package that each version of the network card driver source code depends on from the software package library one by one, compile and install each version of the network card driver source code on a preset operating system, and test at least one third performance indicator value corresponding to each version of the network card driver source code through a network card testing tool.

[0103] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0104] The image file construction device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0105] The embodiment of the present invention also provides a computer device having the above Figure 6 The image file shown is used to build the device.

[0106] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 10 is taken as an example.

[0107] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0108] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0109] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0110] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0111] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0112] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0113] Part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes but is not limited to source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0114] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for constructing an image file, characterized in that: The method comprises: Obtain M versions of driver source code corresponding to a first model, and application software associated with each version of the driver source code, wherein the first model is a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes are all the same; Installing and testing the M versions of the driver source code and the associated application software one by one to obtain a test result, wherein the test result includes at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code; Based on the at least one first performance indicator value and the at least one second performance indicator value, determining a target version among the M versions, and acquiring a target driver source code corresponding to the target version, and a target-associated application software associated with the target driver source code; Constructing a target image file corresponding to the training cluster according to the target driver source code, the target associated application software and a preset file, wherein the preset file represents an associated file of a preset operating system corresponding to the N computing nodes in the training cluster; The target image file is sent to the management node of the training cluster.

2. The method according to claim 1, characterized in that The M versions of the driver source code and the associated application software are installed and tested one by one to obtain test results, including: Installing the preset operating system and configuring a software package library of the preset operating system, wherein the software package library includes a plurality of software packages; The first software package that each version of the driver source code depends on and the second software package that the associated application software depends on are called one by one from the software package library, each version of the driver source code and the associated application software are compiled and installed on the preset operating system, and the at least one first performance indicator value corresponding to the associated application software and the at least one second performance indicator value corresponding to each version of the driver source code are tested through a preset testing tool.

3. The method according to claim 2, characterized in that The at least one first performance indicator value includes the throughput, delay duration and error rate of request processing corresponding to each version of the driver source code, and the at least one second performance indicator value includes the average processor usage, average memory usage and average disk usage corresponding to the associated application software; The determining a target version from the M versions based on the at least one first performance indicator value and the at least one second performance indicator value includes: Obtain a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage, a fifth weight corresponding to the average memory usage, and a sixth weight corresponding to the average disk usage; Obtaining a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software; Calculate the score corresponding to each version of the driver source code based on a preset algorithm, the at least one first performance indicator value, and the at least one second performance indicator value; Determine the version with the highest score among the M versions as the target version; The preset algorithm is: Q=μ*(α1*A+β1*B+γ1*C)+(1-μ)*(α2*D+β2*E+γ2*F) wherein μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents the throughput; B represents the delay duration; C represents the error rate; D represents the average processor utilization; E represents the average memory utilization; F represents the average disk utilization.

4. The method according to claim 3, characterized in that The preset files include the target network card driver source code, the third software package on which the target network card driver source code depends, the first software package, the second software package, the operating system files corresponding to the preset operating system, cluster management related files and resource monitoring related files.

5. The method according to claim 4, characterized in that The step of constructing a target image file corresponding to the training cluster according to the target driver source code, the target associated application software and the preset file includes: Obtaining a first configuration file, wherein the first configuration file is used to indicate a construction order among the target driver source code, the target associated application software, and all files included in the preset file; Based on the first configuration file, the target image file corresponding to the training cluster is constructed in sequence according to the order of the operating system file, the first software package, the second software package, the third software package, the target driver source code, the target associated application software, the target network card driver source code, the cluster management related files and the resource monitoring related files.

6. The method according to claim 5, characterized in that The method further comprises: Obtaining P versions of network card driver source code corresponding to a second model, where the second model is an infinite bandwidth network card model corresponding to the N computing nodes in the training cluster, and the infinite bandwidth network card models corresponding to the N nodes are all the same; Installing and testing the P versions of the network card driver source code one by one to obtain a network card test result, wherein the network card test result includes at least one third performance indicator value corresponding to each version of the network card driver source code; Based on the at least one third performance indicator value, a first version is determined among the P versions, and the target network card driver source code corresponding to the first version is obtained.

7. The method according to claim 6, characterized in that The step of installing and testing the P versions of the network card driver source codes one by one to obtain the network card test results includes: The third software packages that each version of the network card driver source code depends on are called one by one from the software package library, each version of the network card driver source code is compiled and installed on the preset operating system, and at least one third performance indicator value corresponding to each version of the network card driver source code is tested through a network card testing tool.

8. A device for constructing an image file, characterized in that: The device comprises: A transceiver module is used to obtain M versions of driver source code corresponding to a first model, and application software associated with each version of the driver source code, wherein the first model is a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes are all the same; a processing module, configured to install and test the M versions of the driver source code and the associated application software one by one, and obtain a test result, wherein the test result includes at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code; The processing module is further used to determine a target version among the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and obtain a target driver source code corresponding to the target version, and a target-associated application software associated with the target driver source code; The processing module is further used to construct a target image file corresponding to the training cluster according to the target driver source code, the target associated application software and a preset file, wherein the preset file represents an associated file of a preset operating system corresponding to the N computing nodes in the training cluster; The transceiver module is further used to send the target image file to the management node of the training cluster.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image file construction method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image file construction method according to any one of claims 1 to 7.

Images