A mirror file construction method and device, computer equipment and storage medium

Abstract

The application relates to the technical field of computers and discloses a mirror file construction method and device, computer equipment and a storage medium, the method comprising the following steps: acquiring drive source codes of M versions corresponding to a first model and application software associated with each version of the drive source codes; installing and testing the M versions of the drive source codes and the associated application software one by one to obtain test results; determining a target version in the M versions based on at least one first performance index value and at least one second performance index value included in the test results, and acquiring target drive source codes corresponding to the target version and target associated application software; constructing a target mirror file corresponding to a training cluster according to the target drive source codes, the target associated application software and a preset file; and sending the target mirror file to a management node of the training cluster. The application can solve the problems that the training cluster environment deployment process is complex, a large amount of human cost is consumed and abnormalities are prone to occurring in the deployment process.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a mirror file construction method and device, computer equipment and storage medium. BACKGROUND

[0002] With the development of cloud computing and big data technology, artificial intelligence models have not only caused extensive discussion in the academic field due to their huge data throughput and complex algorithm structure, but also shown great potential and value in commercial applications. The challenges of intelligent model training are multifaceted. For example, model training requires huge computing resources. For another example, the construction and deployment process of model training cluster is complex.

[0003] The current model training cluster deployment process is complex, not only requiring the deployment of operating systems, processor drivers, network card drivers and other application software of each computing node, the installation of container components, cluster management components and cluster resource monitoring components, etc., consuming a large amount of labor cost, but also in the process of deployment, problems such as driver incompatibility or network card driver incompatibility, component version dependency, computing node container service startup exception, incomplete operating system configuration, etc. may occur. SUMMARY

[0004] Therefore, the present application provides a mirror file construction method and device, computer equipment and storage medium to solve the problems of complex training cluster environment deployment process, large labor cost and easy occurrence of exceptions in the process of deployment.

[0005] In a first aspect, the present application provides a mirror file construction method, which comprises: obtaining M versions of drive source code corresponding to a first model and application software associated with each version of the drive source code, the first model being a processor model corresponding to N computing nodes in a training cluster, the processor models corresponding to the N nodes being the same; installing and testing the M versions of drive source code and the associated application software one by one to obtain test results, the test results including at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the drive source code; determining a target version in the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and obtaining a target drive source code corresponding to the target version and a target associated application software associated with the target drive source code; constructing a target mirror file corresponding to the training cluster according to the target drive source code, the target associated application software and a preset file, the preset file representing associated files of a preset operating system corresponding to the N computing nodes in the training cluster; and sending the target mirror file to a management node of the training cluster.

[0006] Based on the method of the first aspect, the target driver source code corresponding to the N computing nodes in the training cluster can be obtained, and the target associated application software associated with the target driver source code can be obtained, that is, the processor driver source code required by each computing node in the training cluster and the associated application software can be determined. In addition, the target image file can be constructed according to the target driver source code, the target associated application software, and the associated files of the preset operating system, so that the target image file integrates the processor driver source code required by each computing node in the training cluster and the associated application software, and the associated various files of the preset operating system, that is, the compatibility problem of the processor driver source code and the application software is pre-processed, and the optimal selection problem of the associated various files of the preset operating system. Only the target image file needs to be sent to the management node of the training cluster, that is, the management node can send the target image file to each computing node for training cluster environment deployment, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, improving the deployment efficiency, and avoiding the problems of incompatible driver source code and application software dependency.

[0007] In an optional embodiment, the M versions of the driver source code and the associated application software are installed and tested one by one to obtain test results, including: installing a preset operating system and configuring a software package library of the preset operating system, the software package library including a plurality of software packages; calling a first software package dependent on each version of the driver source code and a second software package dependent on the associated application software from the software package library one by one; compiling and installing each version of the driver source code and the associated application software on the preset operating system; and testing at least one first performance index value corresponding to the associated application software and at least one second performance index value corresponding to each version of the driver source code through a preset test tool.

[0008] In the embodiment of the application, the performance indicators of each version of the driver source code and the associated application software can be tested on the installed preset operating system, and the first software package dependent on each version of the driver source code and the second software package dependent on the associated application software can be obtained, which facilitates subsequent determination of the best performance driver source code based on at least one first performance index value and at least one second performance index value.

[0009] In an optional implementation, the at least one first performance indicator value includes a throughput, a delay duration and an error rate of processing a request corresponding to each version of the driver source code, and the at least one second performance indicator value includes an average processor usage rate, an average memory usage rate and an average disk usage rate corresponding to the associated application software; and the target version is determined from the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, including: obtaining a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage rate, a fifth weight corresponding to the average memory usage rate, and a sixth weight corresponding to the average disk usage rate; and obtaining a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software.

[0010] Based on the preset algorithm, the at least one first performance indicator value and the at least one second performance indicator value, a score value corresponding to each version of the driver source code is calculated; and the version with the highest score value is determined as the target version from the M versions.

[0011] The preset algorithm is Q=μ*(α1*A+β1*B+γ1*C)+(1-μ)*(α2*D+β2*E+γ2*F). Wherein, μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents the throughput; B represents the delay duration; C represents the error rate; D represents the average processor usage rate; E represents the average memory usage rate; and F represents the average disk usage rate.

[0012] In the embodiments of the present application, the score value corresponding to each version of the driver source code can be calculated based on the preset algorithm, the at least one first performance indicator value and the at least one second performance indicator value, and then the target driver source code corresponding to the target version is obtained, so that the most suitable driver source code is selected.

[0013] In an optional implementation, the preset file includes the target network card driver source code, a third software package dependent on the target network card driver source code, the first software package, the second software package, an operating system file corresponding to a preset operating system, a cluster management related file and a resource monitoring related file of a training cluster.

[0014] In the embodiment of the present application, the target image file can be pre-integrated with the target network card driver source code, the third software package dependent on the target network card driver source code, the first software package, the second software package, the operating system file corresponding to the preset operating system, the cluster management related file and the resource monitoring related file of the training cluster, so as to pre-process the compatibility problem of the network card driver source code, the installation dependency problem of the network card driver source code and the target driver source code, and the optimal selection problem of the operating system and various components, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, and improving the deployment efficiency.

[0015] In an optional embodiment, the target image file corresponding to the training cluster is constructed according to the target driver source code, the target associated application software and the preset file, and the method comprises the following steps: obtaining a first configuration file, the first configuration file being used to indicate the construction sequence between all files included in the target driver source code, the target associated application software and the preset file; and based on the first configuration file, the target image file corresponding to the training cluster is constructed in sequence according to the order of the operating system file, the first software package, the second software package, the third software package, the target driver source code, the target associated application software, the target network card driver source code, the cluster management related file and the resource monitoring related file.

[0016] In the embodiment of the present application, the target image file can be constructed according to the construction sequence indicated by the first configuration file, so that the target image file can be directly deployed to the management node and each computing node in the training cluster.

[0017] In an optional embodiment, the method further comprises the following steps: obtaining P versions of network card driver source codes corresponding to a second type, the second type being the type of the InfiniBand network card corresponding to the N computing nodes in the training cluster, and the types of the InfiniBand network cards corresponding to the N nodes being all the same; installing and testing the P versions of network card driver source codes one by one to obtain network card test results, the network card test results comprising at least one third performance index value corresponding to each version of network card driver source code; determining a first version in the P versions based on the at least one third performance index value, and obtaining the target network card driver source code corresponding to the first version.

[0018] In the embodiment of the present application, the target network card driver source code corresponding to the N computing nodes in the training cluster can be obtained, that is, the InfiniBand network card driver source code required by each computing node in the training cluster can be determined.

[0019] In an optional implementation, the P versions of the network card driver source code are installed and tested one by one to obtain the network card test result, including: calling the third software package dependent on each version of the network card driver source code from the software package library one by one, compiling and installing each version of the network card driver source code on the preset operating system, and testing at least one third performance index value corresponding to each version of the network card driver source code through the network card test tool.

[0020] In the implementation of the present application, the performance index of each version of the network card driver source code can be tested on the installed preset operating system, and the third software package dependent on each version of the network card driver source code can be obtained, which facilitates subsequent determination of the best performance network card driver source code based on at least one third performance index value.

[0021] In a second aspect, the present application provides a mirror file construction device, which comprises: a transceiver module configured to obtain M versions of driver source code corresponding to a first model and application software associated with each version of the driver source code, the first model being a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes being all the same; and a processing module configured to install and test the M versions of the driver source code and the associated application software one by one to obtain a test result, the test result comprising at least one first performance index value corresponding to the associated application software and at least one second performance index value corresponding to each version of the driver source code.

[0022] The processing module is further configured to determine a target version from the M versions based on the at least one first performance index value and the at least one second performance index value, and obtain a target driver source code corresponding to the target version and target associated application software associated with the target driver source code; and the processing module is further configured to construct a target mirror file corresponding to the training cluster according to the target driver source code, the target associated application software, and a preset file, the preset file representing associated files of preset operating systems corresponding to the N computing nodes in the training cluster; and the transceiver module is further configured to send the target mirror file to a management node of the training cluster.

[0023] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the mirror file construction method of the first aspect or any of the corresponding embodiments thereof.

[0024] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for causing a computer to perform the mirror file construction method of the first aspect or any of the corresponding embodiments thereof.

[0025] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the image file construction method of the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the training cluster deployment process according to an embodiment of the present invention.

[0028] Figure 2 This is a topology diagram of the image file building system according to an embodiment of the present invention;

[0029] Figure 3 This is a flowchart illustrating the image file construction method according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the target image file installation process according to an embodiment of the present invention;

[0031] Figure 5 This is a structural block diagram of an image file building apparatus according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] This invention applies to scenarios involving the deployment of training cluster environments. In the deployment of a training cluster environment, a problem with any computing node can significantly delay the deployment progress and affect the training performance of the model. As described in the background section, the current process of deploying a model training cluster is complex, such as... Figure 1 As shown, Figure 1 This is a schematic diagram illustrating the training cluster deployment process according to embodiments of the present invention;Figure 1 In this process, the operating system is deployed using Preboot Execution Environment (PXE) technology; drivers are deployed, and their compatibility is checked. If they are incompatible, a suitable driver is selected; if they are compatible, application software is deployed, and any software dependencies are checked. If dependencies exist, a suitable application software is selected; if no dependencies exist, cluster management components are deployed. Once all software is installed, the cluster environment deployment is complete.

[0035] Figure 1 The training cluster deployment method shown has many problems. For example, due to a lack of experience in driver selection, it is difficult to choose the most suitable driver to match the hardware, and various compatibility issues need to be resolved during the adaptation process; how to choose the upper-layer application version, which version has better compatibility with the driver and hardware, and better performance; what software to use to monitor various parameters when using the training cluster; how to manage the cluster and ensure its stability; how to ensure the network stability of each computing node in the training cluster; and how to handle the interaction between each module and the operating system itself, etc.

[0036] To address the aforementioned issues, this invention provides a method for constructing image files. By constructing target image files corresponding to various files required for training cluster environment deployment, the method simplifies the deployment process, reduces deployment costs, and improves deployment efficiency.

[0037] The following is based on Figure 2 Taking the image file building system 200 shown as an example, the method provided in the embodiments of this application is described. Figure 2 This is merely an illustrative diagram and does not constitute a limitation on the applicable scenarios of the technical solutions provided in this application.

[0038] like Figure 2 As shown, Figure 2 This is a topology diagram of an image file building system according to an embodiment of the present invention. Figure 2 In the image file building system 200, the image file building device 201, the management node 202, the first computing node 203, and the second computing node 204 may be included.

[0039] In this embodiment of the invention, the image file building device 201 can be any device with communication and computing capabilities. For example, the image file building device can be a server, a cloud server, or a virtual machine.

[0040] In the embodiment of the present application, the management node 202, the first computing node 203 or the second computing node 204 can be any device with communication and computing capabilities. The management node 202, the first computing node 203 and the second computing node 204 belong to the nodes in the training cluster. The management node 202 is used to manage and monitor the first computing node 203 and the second computing node 204.

[0041] Figure 2 The mirror file construction system 200 shown is only used for example and does not limit the technical solutions of the present application. Those skilled in the art should understand that in the specific implementation process, the mirror file construction system 200 can also include other devices, and the number of computing nodes can also be determined according to specific needs, and is not limited.

[0042] According to the embodiment of the present application, a mirror file construction method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0043] In the present embodiment, a mirror file construction method is provided, which can be used for the mirror file construction device described above, Figure 3 is a flowchart of the mirror file construction method according to the embodiment of the present application, as Figure 3 shown, the flow includes the following steps:

[0044] S301: Obtain the drive source code of M versions corresponding to the first model, and the application software associated with each version of the drive source code.

[0045] Among them, the first model is the processor model corresponding to the N computing nodes in the training cluster, and the processor models corresponding to the N nodes are all the same. Optionally, the processor can be a graphics processing unit (GPU).

[0046] It can be understood that a model of processor can correspond to multiple versions of drive source code. Different drive source codes are associated with different application software. The associated application software can be one or more application software.

[0047] S302: Install and test the M versions of the drive source code and the associated application software one by one to obtain the test results.

[0048] Among them, the test results include at least one first performance index value corresponding to the associated application software and at least one second performance index value corresponding to each version of the drive source code.

[0049] In some optional embodiments, the mirror file construction device installs a preset operating system and configures a software package library of the preset operating system; calls first software packages dependent on each version of the driver source code and second software packages dependent on the associated application software from the software package library one by one, compiles and installs each version of the driver source code and the associated application software on the preset operating system, and tests at least one first performance index value corresponding to the associated application software and at least one second performance index value corresponding to each version of the driver source code by using a preset test tool.

[0050] The software package library includes a plurality of software packages. The software package library can be a yum source. The software package library refers to a repository for storing and managing a software package manager (RPM).

[0051] The preset operating system can be a pure operating system.

[0052] In the embodiments of the present application, the at least one first performance index value includes throughput, delay duration, and error rate of processing requests corresponding to each version of the driver source code.

[0053] In the embodiments of the present application, the at least one second performance index value includes average processor usage, average memory usage, and average disk usage corresponding to the associated application software.

[0054] In the embodiments of the present application, the associated application software can be a deep neural network library (cuDNN), a container (docker), a collective communication library (NCCL), and a network manager (nvidia-fabric-manager).

[0055] In the embodiments of the present application, the preset test tool can be a driver test tool and an application software test tool. For example, the driver test tool can be a driver verifier (Driver Verifier). The application software test tool can be a load runner (LoadRunner) and a debugger (GNU Debugger).

[0056] It can be understood that in the process of installing and testing the M versions of the driver source code and the associated application software one by one by the mirror file construction device, the plurality of software packages included in the software package library can be called.

[0057] S303: determining a target version from the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and obtaining a target drive source code corresponding to the target version and a target associated application software associated with the target drive source code.

[0058] In some optional embodiments, the mirror file construction apparatus obtains a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage, a fifth weight corresponding to the average memory usage, and a sixth weight corresponding to the average disk usage; obtains a first coefficient corresponding to the drive source code of each version and a second coefficient corresponding to the associated application software; calculates a score value corresponding to the drive source code of each version based on a preset algorithm, the at least one first performance indicator value, and the at least one second performance indicator value; and determines a version with the highest score value as the target version from the M versions.

[0059] The preset algorithm is Q = μ * (α1*A + β1*B + γ1*C) + (1-μ) * (α2*D + β2*E + γ2*F).

[0060] μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents the throughput; B represents the delay duration; C represents the error rate; D represents the average processor usage; E represents the average memory usage; and F represents the average disk usage.

[0061] It can be understood that the performance of the drive source code of the processor is not only related to the performance of the drive source code itself, but also related to the performance of the application program associated with the drive source code. Therefore, the score value corresponding to the drive source code of each version can be determined based on the first coefficient corresponding to the drive source code of each version and the second coefficient corresponding to the associated application software.

[0062] S304: constructing a target mirror file corresponding to the training cluster according to the target drive source code, the target associated application software, and a preset file.

[0063] The preset file represents associated files of a preset operating system corresponding to N computing nodes in the training cluster. The preset file includes the target network card drive source code, a third software package dependent on the target network card drive source code, the first software package, the second software package, an operating system file corresponding to the preset operating system, a cluster management related file of the training cluster, and a resource monitoring related file.

[0064] The cluster management related files and the resource monitoring related files can be a DataCenter GPU Manager (DCGM), a Parallel Distributed Shell (PDSh), and an openLightweight Directory Access Protocol (openldap) service.

[0065] In some optional embodiments, the image file construction apparatus obtains a first configuration file; and based on the first configuration file, sequentially constructs the target image file corresponding to the training cluster in the order of the operating system file, the first software package, the second software package, the third software package, the target driver source code, the target associated application software, the target network card driver source code, the cluster management related files, and the resource monitoring related files.

[0066] The first configuration file is used to indicate the construction order between the target driver source code, the target associated application software, and all files included in the preset file.

[0067] It can be understood that the operating system file, the first software package, the second software package, the third software package, the target driver source code, the target associated application software, the target network card driver source code, the cluster management related files, and the resource monitoring related files have a specific installation order. If the target image file is not constructed according to the installation order, the deployment of the training cluster will fail or an exception will occur. For example, before installing the DCGM, the network manager (nvidia-fabric-manager) and the software package related to the compiler set (GNU Compiler Collection, GCC) required for compiling the driver need to be installed.

[0068] In some optional embodiments, the image file construction apparatus obtains P versions of the network card driver source code corresponding to the second model; installs and tests the P versions of the network card driver source code one by one to obtain a network card test result, the network card test result including at least one third performance index value corresponding to each version of the network card driver source code; based on the at least one third performance index value, determines a first version in the P versions, and obtains a target network card driver source code corresponding to the first version.

[0069] The second model is a type of InfiniBand network card corresponding to N computing nodes in the training cluster, and the InfiniBand network cards corresponding to the N nodes are of the same type. The InfiniBand network card is also referred to as an InfiniBand Card (IB) card.

[0070] In one example, the image file building device calls the third software packages that each version of the network card driver source code depends on from the software package library one by one, compiles and installs each version of the network card driver source code on a preset operating system, and tests at least one third performance index value corresponding to each version of the network card driver source code through a network card testing tool.

[0071] Among them, at least one third performance metric includes the network card throughput, network card latency, number of concurrent connections, and connection establishment time for each version of the network card driver source code.

[0072] In this embodiment of the invention, the network interface card (NIC) testing tool can be a custom script, a load testing tool, etc.

[0073] In one example, the image file building device obtains the seventh weight corresponding to the network card throughput, the eighth weight corresponding to the network card latency, the ninth weight corresponding to the number of concurrent connections, and the tenth weight corresponding to the connection establishment time. Based on the first algorithm, the network card throughput, the network card latency, the number of concurrent connections, and the connection establishment time, it calculates the network card score corresponding to each version of the network card driver source code, and determines the version with the highest network card score among P versions as the first version.

[0074] The first algorithm is: S = α3*H + β3*I + γ3*J + θ*K.

[0075] Where α3 represents the seventh weight; β3 represents the eighth weight; γ3 represents the ninth weight; θ represents the tenth weight; H represents the network interface card throughput; I represents the network interface card latency; J represents the number of concurrent connections; and K represents the connection establishment time.

[0076] Understandably, since network card throughput, network card latency, concurrent connections and connection establishment time have different impacts on the performance of network card driver source code, the first algorithm can be used to accurately determine the network card score corresponding to each version of the network card driver source code, and then select the best target network card driver source code.

[0077] S305: Send the target image file to the management node of the training cluster.

[0078] The target image file may contain parameter configuration information.

[0079] For example, the image file building device sends the target image file to the management node of the training cluster. The management node receives the target image file and then sends it to each compute node in the training cluster.

[0080] like Figure 4 As shown, Figure 4 This is a schematic diagram of the target image file installation process according to an embodiment of the present invention; the management node and each compute node can perform the following:

[0081] S401: installing the target image file through the PXE technology.

[0082] S402: monitoring the installation progress of the target image file through the monitoring tool on the management node.

[0083] S403: judging whether the target image file is installed completely based on the installation progress; if not, continuing the installation.

[0084] S404: if installed completely, restarting and configuring each module included in the target image file based on the parameter configuration information, and the training cluster deployment is completed.

[0085] Based on the method, Figure 3 the image file construction device can obtain the target driver source code corresponding to the N computing nodes in the training cluster and the target associated application software associated with the target driver source code, that is, the processor driver source code required by each computing node in the training cluster and the associated application software can be determined. In addition, the target image file can be constructed according to the target driver source code, the target associated application software and the associated files of the preset operating system, so that the target image file integrates the processor driver source code required by each computing node in the training cluster and the associated application software, and the associated various files of the preset operating system, that is, the processor driver source code and the application software compatibility problem, the best selection problem of the associated various files of the preset operating system are pre-processed. Only the target image file needs to be sent to the management node of the training cluster, that is, the management node can send the target image file to each computing node for training cluster environment deployment, thereby simplifying the training cluster environment deployment process, reducing the deployment cost of the training cluster environment deployment, improving the deployment efficiency, and avoiding the problems of driver source code incompatibility and application software dependency.

[0086] In the embodiment, an image file construction device is also provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is contemplated.

[0087] The embodiment provides an image file construction device, as shown in Figure 5 , Figure 5 is a structural block diagram of the image file construction device according to the embodiment of the application; the device comprises:

[0088] The transceiving module 501 is configured to acquire M versions of driver source code corresponding to a first model and application software associated with each version of the driver source code, the first model being a processor model corresponding to N computing nodes in a training cluster, and the processor models corresponding to the N nodes being all the same.

[0089] The processing module 502 is configured to install and test the M versions of driver source code and the associated application software one by one to obtain a test result, the test result including at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code.

[0090] The processing module 502 is further configured to determine a target version from the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and acquire target driver source code corresponding to the target version and target associated application software associated with the target driver source code.

[0091] The processing module 502 is further configured to construct a target image file corresponding to the training cluster according to the target driver source code, the target associated application software, and a preset file, the preset file representing associated files of a preset operating system corresponding to the N computing nodes in the training cluster.

[0092] The transceiving module 502 is further configured to send the target image file to a management node of the training cluster.

[0093] In some optional embodiments, the processing module 502 is specifically configured to install a preset operating system and configure a software package library of the preset operating system, the software package library including a plurality of software packages; call a first software package dependent on each version of the driver source code and a second software package dependent on the associated application software from the software package library one by one; compile and install each version of the driver source code and the associated application software on the preset operating system; and test, through a preset test tool, the at least one first performance indicator value corresponding to the associated application software and the at least one second performance indicator value corresponding to each version of the driver source code.

[0094] In some optional embodiments, the at least one first performance indicator value includes throughput, delay duration, and error rate of processing requests corresponding to each version of the driver source code, and the at least one second performance indicator value includes average processor usage rate, average memory usage rate, and average disk usage rate corresponding to the associated application software.

[0095] The processing module 502 is also specifically configured to obtain a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage rate, a fifth weight corresponding to the average memory usage rate, and a sixth weight corresponding to the average disk usage rate; obtain a first coefficient corresponding to the driving source code of each version and a second coefficient corresponding to the associated application software; calculate a score value corresponding to the driving source code of each version based on a preset algorithm, at least one first performance indicator value, and at least one second performance indicator value; and determine, among the M versions, a version with the highest score value as the target version.

[0096] The preset algorithm is:

[0097] Q = μ * (α1*A + β1*B + γ1*C) + (1-μ) * (α2*D + β2*E + γ2*F)

[0098] wherein μ represents the first coefficient; (1-μ) represents the second coefficient; α1 represents the first weight; β1 represents the second weight; γ1 represents the third weight; α2 represents the fourth weight; β2 represents the fifth weight; γ2 represents the sixth weight; A represents the throughput; B represents the delay duration; C represents the error rate; D represents the average processor usage rate; E represents the average memory usage rate; and F represents the average disk usage rate.

[0099] In some optional embodiments, the preset file includes the target network card driving source code, a third software package dependent on the target network card driving source code, the first software package, the second software package, an operating system file corresponding to a preset operating system, a cluster management related file, and a resource monitoring related file.

[0100] In some optional embodiments, the processing module 502 is also specifically configured to obtain a first configuration file, the first configuration file being used to indicate a construction sequence between the target driving source code, the target associated application software, and all files included in the preset file; and based on the first configuration file, sequentially construct the target image file corresponding to the training cluster in the order of the operating system file, the first software package, the second software package, the third software package, the target driving source code, the target associated application software, the target network card driving source code, the cluster management related file, and the resource monitoring related file.

[0101] In some optional embodiments, the transceiving module 501 is further configured to obtain P versions of network card driver source code corresponding to a second model, the second model being a model of an InfiniBand network card corresponding to N computing nodes in the training cluster, the InfiniBand network cards corresponding to the N nodes all being of the same model; the processing module 502 is further configured to install and test the P versions of network card driver source code one by one to obtain network card test results, the network card test results including at least one third performance index value corresponding to each version of network card driver source code; and the processing module 502 is further configured to determine a first version from the P versions based on the at least one third performance index value and obtain target network card driver source code corresponding to the first version.

[0102] In some optional embodiments, the processing module 502 is further configured to call third software packages dependent on each version of network card driver source code from a software package library one by one, compile and install each version of network card driver source code on a preset operating system, and test at least one third performance index value corresponding to each version of network card driver source code by using a network card test tool.

[0103] Further function descriptions of the above-mentioned various modules and units are the same as those of the corresponding embodiments, and will not be described here again.

[0104] The mirror file construction apparatus in the embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory executing one or more software or fixed programs, and / or other devices that can provide the above functions.

[0105] The embodiment of the present application further provides a computer device having the above mirror file construction apparatus. Figure 6 as shown in the mirror file construction apparatus.

[0106] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of a computer device provided by an optional embodiment of the present application, as shown in Figure 6As shown, the computer device includes one or more processors 10, memory 20, and interfaces 30 for the various components to communicate with one another. The various components communicate through the use of the various buses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 10 can process instructions for execution within the computer device, including instructions stored in the memory 20 or elsewhere to implement routines for displaying graphical information, such as a GUI on an external input / output device, such as a display device coupled to the interface 30. In some embodiments, multiple processors and / or multiple buses can be employed as appropriate, as will be appreciated by those skilled in the art. Additionally, various components of the computer device can be used for other purposes, such as storage of data to implement routines, as will also be appreciated by those skilled in the art. Figure 6 The processor 10 is taken as an example in the embodiments.

[0107] The processor 10 can be a central processing unit, a network processor, or a combination thereof. The processor 10 can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic device, a general array logic, or any combination thereof.

[0108] The memory 20 stores instructions that are executable by the at least one processor 10, so as to enable the at least one processor 10 to perform the method shown in the above embodiments.

[0109] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory that is remotely arranged with respect to the processor 10, and these remote memories can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0110] The memory 20 can include a volatile memory, such as a random access memory, and can also include a non-volatile memory, such as a flash memory, a hard disk, or a solid state disk. The memory 20 can further include a combination of the above-mentioned kinds of memories.

[0111] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or communication networks.

[0112] The embodiments of the present application further provide a computer readable storage medium, and the method according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network, so that the method described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, etc. Further, the storage medium can also include a combination of the above-mentioned memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0113] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in a computer readable medium includes but is not limited to source files, executable files, installation package files, etc. Correspondingly, the way of executing computer program instructions by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0114] Although the embodiments of the present application are described in conjunction with the accompanying drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A mirror file construction method, characterized by, The method comprises: obtaining M versions of driver source code corresponding to a first model, and application software associated with each version of the driver source code, the first model being a processor model corresponding to N computing nodes in a training cluster, the processor models corresponding to the N computing nodes being all the same; installing and testing the M versions of driver source code and the associated application software one by one to obtain test results, the test results including at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the driver source code; determining a target version in the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and obtaining target driver source code corresponding to the target version and target associated application software associated with the target driver source code; constructing a target image file corresponding to the training cluster according to the target driver source code, the target associated application software, and a preset file, the preset file representing associated files of a preset operating system corresponding to the N computing nodes in the training cluster, the preset file including target network card driver source code, a third software package dependent on the target network card driver source code, a first software package, a second software package, operating system files corresponding to the preset operating system, cluster management related files, and resource monitoring related files; sending the target image file to a management node of the training cluster.

2. The method of claim 1, wherein, The installing and testing the M versions of driver source code and the associated application software one by one to obtain test results comprises: installing the preset operating system and configuring a software package library of the preset operating system, the software package library including a plurality of software packages; calling the first software package dependent on each version of the driver source code and the second software package dependent on the associated application software from the software package library one by one, compiling and installing each version of the driver source code and the associated application software on the preset operating system, and testing the at least one first performance indicator value corresponding to the associated application software and the at least one second performance indicator value corresponding to each version of the driver source code through a preset test tool.

3. The method of claim 2, wherein, The at least one first performance indicator value includes throughput, delay duration, and error rate of processing requests corresponding to each version of the driver source code, and the at least one second performance indicator value includes average processor usage rate, average memory usage rate, and average disk usage rate corresponding to the associated application software; The determining a target version in the M versions based on the at least one first performance indicator value and the at least one second performance indicator value comprises: obtaining a first weight corresponding to the throughput, a second weight corresponding to the delay duration, a third weight corresponding to the error rate, a fourth weight corresponding to the average processor usage rate, a fifth weight corresponding to the average memory usage rate, and a sixth weight corresponding to the average disk usage rate; obtaining a first coefficient corresponding to each version of the driver source code and a second coefficient corresponding to the associated application software; calculating a score value corresponding to each version of the drive source code based on a preset algorithm, the at least one first performance indicator value, and the at least one second performance indicator value; determining, among the M versions, a version with the highest score value as the target version; the preset algorithm is: wherein, denotes the first coefficient; denotes the second coefficient; denotes the first weight; denotes the second weight; denotes the third weight; denotes the fourth weight; denotes the fifth weight; denotes the sixth weight; denotes the throughput; denotes the latency; denotes the error rate; denotes the average processor usage; denotes the average memory usage; denotes the average disk usage.

4. The method of claim 3, wherein, the target image file corresponding to the training cluster is constructed according to the target drive source code, the target associated application software, and a preset file, including: a first configuration file is obtained, which is used to indicate the construction sequence between all files included in the target drive source code, the target associated application software, and the preset file; based on the first configuration file, the target image file corresponding to the training cluster is constructed in the order of the operating system file, the first software package, the second software package, the third software package, the target drive source code, the target associated application software, the target network card drive source code, the cluster management related file, and the resource monitoring related file.

5. The method of claim 4, wherein, The method further includes: obtaining P versions of network card drive source code corresponding to a second model, the second model being an infiniband network card model corresponding to the N computing nodes in the training cluster, the infiniband network card models corresponding to the N computing nodes being all the same; installing and testing the P versions of network card drive source code one by one to obtain network card test results, the network card test results including at least one third performance indicator value corresponding to each version of the network card drive source code; based on the at least one third performance indicator value, determining a first version among the P versions, and obtaining the target network card drive source code corresponding to the first version.

6. The method of claim 5, wherein, The installing and testing the P versions of network card drive source code one by one to obtain network card test results includes: calling the third software package dependent on each version of the network card drive source code from the software package library one by one, compiling and installing each version of the network card drive source code on the preset operating system, and testing the at least one third performance indicator value corresponding to each version of the network card drive source code through a network card test tool.

7. An image file construction apparatus characterized by comprising: The device includes: a transceiver module configured to obtain M versions of drive source code corresponding to a first model, and application software associated with each version of the drive source code, the first model being a processor model corresponding to N computing nodes in a training cluster, the processor models corresponding to the N computing nodes being all the same; a processing module configured to install and test the M versions of drive source code and the associated application software one by one to obtain test results, the test results including at least one first performance indicator value corresponding to the associated application software and at least one second performance indicator value corresponding to each version of the drive source code; the processing module is further configured to determine a target version among the M versions based on the at least one first performance indicator value and the at least one second performance indicator value, and obtain a target drive source code corresponding to the target version and a target associated application software associated with the target drive source code; The processing module is further configured to construct a target image file corresponding to the training cluster according to the target driver code, the target associated application software, and a preset file, the preset file representing associated files of a preset operating system corresponding to the N computing nodes in the training cluster, and the preset file including a target network card driver code, a third software package dependent on the target network card driver code, a first software package, a second software package, operating system files corresponding to the preset operating system, cluster management related files, and resource monitoring related files. The transceiving module is further configured to send the target image file to a management node of the training cluster.

8. A computer device, comprising: The transceiving module is further configured to send the target image file to a management node of the training cluster. The memory and the processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the image file construction method in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to perform the image file construction method in any one of claims 1 to 6.

Images