Method and system for launching applications in a simulated environment

Abstract

The invention relates to computing. A method for launching an application in a simulated environment comprises the steps of: receiving an application launch request containing a configuration for the computing resources (CR) required to launch the application. On the basis of the CR configuration, components simulating the CRs are deployed. On the basis of the CR configuration, a simulated server is deployed which contains replacement libraries for the transmission of commands to the simulated CRs. The CR configuration and the application are placed in an isolated environment, a program code is executed in said isolated environment in accordance with the CR configuration, and requests addressed to the CRs are forwarded to the CR-simulating components. Responses to the requests are generated by the CR-simulating components, the responses to said requests are returned to the isolated environment, and execution of the program code in the isolated environment is finished. The technical result consists in making it possible to launch an application on a computing device that is not equipped with the appropriate computing resources to launch the application in question.

Description

METHOD AND SYSTEM FOR LAUNCHING APPLICATIONS IN A SIMULATED ENVIRONMENT AREA OF TECHNOLOGY

[0001] The presented technical solution generally relates to the field of computing technology, and in particular to the method and device for launching applications in a simulated environment. The presented technical solution allows launching applications that use computations on computing resources (CR), for example, GPU and data exchange between them in a simulated environment without using real CR, collecting statistics on network transmissions and video memory usage. First of all, the presented technical solution is aimed at simulating the training of large neural networks on large clusters to check the correctness of the training code and the subsequent possible assessment of the effectiveness of the network topology. LEVEL OF TECHNOLOGY

[0002] Debugging modern ML applications running on large computing clusters is a well-known challenge for existing computing devices. The challenges are related to the fact that: - the availability of the full amount of computing resources at the debugging stage is limited. - training modern neural networks, in particular large language models, requires huge resources. The GPT-3 neural network, a representative of the class of large neural networks, has 175 billion parameters. Its training required calculations on 1024 NVIDIA TESLA A100 video cards for 34 days, subsequent models will require even more resources. Assembling and setting up a cluster of this size takes a significant amount of time, the ability to debug the application before its completion will be useful. - the cost of a user error is very high, since allocating resources for a full neural network takes a lot of time, and the error may also appear at late stages of program execution. As a result, each debugging run consumes a significant amount of computing resources. - for a specific implementation of a neural network, it is difficult to estimate the resulting load on the communication channels of cluster nodes, taking into account their heterogeneity. However, load data is useful when designing and configuring network infrastructure. - for a specific implementation of a neural network, it is difficult to pre-estimate the consumption of video memory. Information on video memory consumption is necessary for optimizing the training code, in particular for selecting parallelism parameters.

[0003] Also of interest is the task of optimizing the cluster hardware set for a specific class of applications, taking into account the bandwidth and delays of communication channels, their topology and configuration.

[0004] Model cluster The standard method used both in scientific papers and in practice is to run a real application on a small computing cluster with all the necessary hardware resources (CPU, GPU, RAM). This allows you to estimate both the network load and the expected computation time.

[0005] However, this approach faces some limitations: - access to the number of video cards required to store the neural network is required throughout the testing. For example, training the GPT3 neural network described earlier requires at least 40 NVIDIA A100 video cards, which imposes a limit on the minimum cluster size. - differences in the behavior of libraries with different numbers of GPUs. In particular, the popular DeepSpeed ​​library in ZeRO mode evenly distributes some neural network parameters across all GPUs, which means that video memory consumption and the amount of data transferred over the network will differ from a full-size launch. - impossibility of use for small tasks, since for small teams renting even a small number of video cards represents significant expenses. Small tasks can already reach the limit of the minimum number of video cards, not allowing to create a cluster of an even smaller size.

[0006] The prior art discloses methods and systems related to autonomously determining the scope of cloud nodes for big data machine learning use cases, disclosed in application US20210174248A1, published on 21.06.2022, which can be considered as the closest analogue to the presented technical solution. In particular, this document discloses an automated tool, method and system for determining the scope of application, which for each of several combinations of parameter values ​​(i) establish a combination of parameter values ​​describing the use case, (ii) perform a machine learning application training according to a combination of parameter values ​​in the target cloud environment and (iii) measure the computational cost of running the machine learning application. A recommendation regarding the CPU, GPU, and memory configuration for the target cloud environment to run the machine learning application is generated based on the measured computational cost.

[0007] Also, as the closest analogue, the methods for separating data pipeline transfer tasks from the execution flow of a high-performance computing task (for example, distributed deep learning of a model) in a distributed computing system, disclosed in application US20200174840A1, published 04.06.2020, can be considered.The known method includes the steps of receiving a client request for preparing resources for executing a computing task, preparing accelerator resources of one or more accelerator server nodes in a distributed computing system for executing tasks associated with the flow of executing the computing task, and preparing logical nodes in the distributed computing system for creating a data flow pipeline, which is configured to execute data flow operations associated with the computing task for providing data to prepared accelerator resources for executing tasks associated with the flow of executing the computing task. Data flow operations include, for example, data storage input / output operations, data preprocessing operations and intermediate data operations that are separated from the flow of executing the computing task.

[0008] The disadvantage of known solutions is that they require the availability of all the computing resources necessary to run the application. ESSENCE OF THE TECHNICAL SOLUTION

[0009] The technical problem or task posed in this technical solution is to create a simple and reliable method and device for launching applications in a simulated environment.

[0010] The technical result achieved by solving the above-mentioned technical problem is to ensure the possibility of launching an application on a computing device that is not equipped with the appropriate computing resources for launching the said application.

[0011] The specified technical result is achieved by implementing a method for launching an application in a simulated environment, performed by means of at least one computing device, containing the steps of: - receive a request to launch an application, containing the configurations of computing resources (CR) required to launch the application; - based on the VR configuration, components simulating VR are deployed; - based on the VR configuration, at least one simulated server is deployed, containing replacement libraries intended for transmitting commands addressed to the VR to the simulated VRs, with each simulated server corresponding to an isolated environment; - place the VR configurations and the application in at least one isolated environment; - execute the program code in at least one isolated environment in accordance with the VR configuration; - using replacement libraries, redirect requests addressed to VR to components simulating VR; - generate responses to the mentioned requests addressed to the VR by processing them by components simulating the VR; - return the generated responses to the said requests to at least one isolated environment; - terminate the execution of program code in an isolated environment.

[0012] In one of the particular examples of the implementation of the method, the VR configurations contain: - description of video card models, presented as a set of parameters; - description of network card models, presented as a set of parameters; - information about the number of simulated servers and a set of parameters for each simulated server; - network connection parameters; - application to launch.

[0013] In another particular example of implementing the method, the execution of the program code is carried out in at least two isolated environments, wherein the transfer of data from an instance of the application executed in the first isolated environment to an instance of the application executed in the second isolated environment is carried out via a virtual network, if for The mentioned data transfer requires the use of network resources using the functionality of the operating system.

[0014] In another particular example of implementing the method, requests addressed to the VR contain at least one of: VR parameters characterizing the VR ID, the function ID, and the number of required resources.

[0015] In another particular example of implementing the method, the components simulating the VR generate responses to the said requests based on stored information about the state of the simulated VR and the VR parameters contained in the request.

[0016] In another particular example of the implementation of the method, the received requests and the state of the simulated VRs are recorded in a log.

[0017] In another particular example of implementing the method, an additional step of updating the state of the simulated VRs is performed after generating a response to the mentioned requests addressed to the VRs.

[0018] In another particular example of implementing the method, additional steps are performed in which: - deploy an RPC server; - place the address of the RPC server in at least one isolated environment; wherein said redirection of requests and generated responses is performed using the RPC server.

[0019] In another preferred embodiment of the claimed solution, a system for launching applications in a simulated environment is presented, containing at least one computing device and at least one memory containing machine-readable instructions that, when executed by at least one computing device, perform the above-mentioned method. BRIEF DESCRIPTION OF DRAWINGS

[0020] The features and advantages of the present invention will become apparent from the following detailed description of the invention and the accompanying drawings, in which: Fig. 1 shows an example of the implementation of the information processing system. Fig. 2 shows an example of the general appearance of the computing device. DETAILED DESCRIPTION OF THE INVENTION

[0021] Below we will describe the concepts and terms necessary for understanding this technical solution.

[0022] In this technical solution, the term “system” means, among other things, a computer system, a computer (electronic computer), a numerical control (CNC), a PLC (programmable logic controller), computerized control systems and any other devices capable of performing a given, clearly defined sequence of operations (actions, instructions).

[0023] A command processing unit is an electronic unit or integrated circuit (microprocessor) that executes machine instructions (programs).

[0024] The command processing unit reads and executes machine instructions (programs) from one or more storage devices. Storage devices may include, but are not limited to, hard disk drives (HDD), flash memory, ROM (read-only memory), solid-state drives (SSD), and optical drives.

[0025] A program is a sequence of instructions intended for execution by a computer control device or a command processing device.

[0026] A signal is a material embodiment of a message for use in transmitting, processing and storing information.

[0027] A logical element is an element that implements certain logical dependencies between input and output signals. Logical elements are usually used to construct logical circuits of computers, discrete circuits of automatic control and management. All types of logical elements, regardless of their physical nature, are characterized by discrete values ​​of input and output signals.

[0028] In accordance with the diagram shown in Fig. 1, the information processing system 100 comprises a platform manager 10, a user device 20, at least one isolated environment execution server 30, and at least one simulator server 40 connected by communication channels. The communication channels can be implemented on the basis of well-known wired and wireless communication technologies to ensure data exchange, and the data transmission along the said channels is carried out by means of signals, for processing in which the elements of system 100 sending or receiving data are equipped with signal conversion means, such as digital-to-analog converters, analog-to-digital converters, etc.

[0029] The platform manager 10 can be implemented on the basis of widely known computing devices and is equipped with: a server management module 11, a data collection module 12 and at least one communication interface (13 or 14). The said modules can be, for example, implemented on the basis of transistor logic elements placed on a single printed circuit board.

[0030] Manager 10 controls the operation of servers 30 and 40, being responsible for initialization, configuration and shutdown. It is with manager 10 that user device 20 interacts. Console interface 13 can be used as the main interface for communication with device 20. Program interface 14 for Python language can be used as an alternative interface, which provides limited possibilities for configuration, but may be more convenient for some users. Also, this interface 14 allows creating configuration files in a simpler way than using a configuration generator or manual writing due to the brevity of the description and automatic management of the configuration file.

[0031] The user device 20 can be implemented on the basis of at least one computing device equipped with data transmission means and represent, for example, a portable or stationary computer, a mobile phone or smartphone, a tablet, etc.

[0032] The server 30 can be implemented on the basis of widely known computing devices and is equipped with: at least one isolated environment (31, 32 or 33) and a data exchange module 34. The said elements of the server 30 can be, for example, implemented on the basis of logical elements on transistors, placed on a single printed circuit board. The function of isolating the environment can be implemented by means of one of the widely known technologies, such as virtual machines or containers, or achieved by using differentiated computing devices as modules 31-33, connected to the data exchange module 34, wherein the devices 31-34 in turn can be implemented by known methods on the basis of logical elements.

[0033] The server 40 can be implemented on the basis of widely known computing devices and is equipped with: a module 41 for receiving remote calls (RPC), a component simulation module 42, a launch module 43, a log management module 44 and a data storage module 45. The specified elements of the server 40 can be, for example, implemented on the basis of logical elements on transistors placed on a single printed circuit board.

[0034] At the first stage, the user of the device 20 generates a request to launch the application, containing the configurations of the computing resources (CR) required to launch the application. For example, the user can generate a request to launch the program code intended for training the GPT3 neural network using the Megatron LM framework, executed using 16 computing devices, each equipped with 4 video accelerators.

[0035] The generated request for launching the application is then transmitted by the user's command from the device 20 to the manager 10 and via at least one interface 13 or 14 is sent to the data collection module 12. The VR configurations may, for example, contain: - description of video card models, presented as a set of parameters, where each parameter characterizes the video card model, memory capacity, and parameters that the CUDA API can request from the video card. For example, a set of parameters may contain the following data: Nvidia A100 video card, contains 80 GB of video memory, the compute capability parameter is 8.6; - description of network card models, presented as a set of parameters. Among the parameters are the number of ports, the permissible transmission speed. For example, a set of parameters may contain the following data: Mellanox ConnectX-6 has 2 ports and supports communication at a speed of 200 Gbit / s; - information about the number of simulated servers and a set of parameters for each simulated server, with each simulated server requiring the launch of one isolated environment in which one instance of the application will be hosted and launched. The parameters include the server ID, the number and models of connected video cards and network cards, the topology of connections within the server, and the throughput of these connections. For example, a set of parameters may contain the following data: the cluster contains 8 servers, each of which contains 4 A100 video cards and 2 Mellanox ConnectX-6 network cards, the video cards are connected to the NVLink Switch using NVLink technology, with a throughput of 450 GB / s, and each video card and network card connected to the processor using PC1e 4.0 technology with a bandwidth of 31.5 GB / s; - parameters of network connections between simulated components. Each connection is characterized, among other things, by its throughput. For example, all servers are connected to one router using Ethernet technology with a throughput of 200 Gbps; - an application to run, for example, presented in the form of program code or a link to a container image with the application.

[0036] Then, the server management module 11 extracts the VR configurations from the module 12 and sends them together with a command for deploying the components simulating the VR to at least one simulator server 40 (depending on the VR configuration), which are received in the launch module 43. In accordance with the received command, the module 43 deploys the RPC server on the module 41 and the components in the module 42 simulating the VR using known methods. After the said RPC server and components have been deployed, the module 43 sends the address of the RPC server to the module 11.

[0037] Then the manager 1 proceeds to the stage of deploying the simulated servers. In particular, the module 11 sends a command to all instances of the isolated environment execution server 30 for deploying the simulated servers in accordance with the VR configuration (a set of parameters for each simulated server) on the said server 30, wherein each simulated server corresponds to an isolated environment (Fig. 1 shows a variant with 3 simulated environments: 31, 32 and 33), in which replacement libraries are located, for example, CUDA ecosystems, intended for transmitting commands addressed to the VR to the simulated VRs, by sending requests to the RPC server. Each isolated environment is separated from other server components by means of isolation used in the infrastructure, and interacts with other isolated environments by means of data exchange module 34. Isolation can be achieved, for example, using docker technology.In this case, the mentioned module 34 can be implemented on the basis of docker bridge tools or another virtual network.

[0038] After the number of simulated servers requested by the user is deployed, module 11 has information about the addresses of the RPC server and all isolated environments. Module 11 sends at least one server 30 of execution of isolated environments of the VR configuration, the application and the addresses of the RPC server and all isolated environments. The received data are placed by the server 30 in at least one of these isolated environments and, in a particular case, are placed in each isolated environment (31, 32, and 33).

[0039] Then module 11 sends a command to server 30 to launch copies of the application in isolated environments, and server 30 launches the application in isolated environments by launching (executing) the program code in accordance with the VR configuration. If the program code executed in the environment (for example, 31) instructs server 30 to use network resources using the functionality of the operating system to transmit data packets to an application instance located in another environment (for example, 32), then server 30 uses data exchange module 34, which implements the functions of a virtual network to transmit the said data packets from environment 31 to environment 32. For example, if server 30 is located in a Kuberneties cluster using Docker for containerization, module 34 can be implemented based on Docker Bridge.

[0040] In the event that the program code executed in the environment 31 instructs the server 30 to use the VR (for example, instructs to allocate memory using CUDA Runtime, to perform matrix multiplication using CuBLAS, or to exchange data between video cards using NCCL), then the server 30 accesses the replacement libraries intended for transmitting commands addressed to the VR to the simulated VRs, in particular to the RPC server in accordance with the RPC server address stored in the isolated environment, by sending the corresponding request. Requests addressed to the VR may contain VR parameters characterizing, for example, the VR ID, the function ID, the amount of required resources, etc.

[0041] For example, when starting the Megatron LM application, the executable program code may contain a command to allocate memory on the video card by calling the cudaMalloc function, for which the server 30 provides the video card identifier and the required memory size. Accordingly, to call the cudaMalloc function, the server 30 refers to a replacement library, which, in turn, by the remote call mechanism, redirects the request to call the cudaMalloc function to the RPC server, transmitting the video card identifier and the required memory size.

[0042] The said request addressed to the VR is received by the RPC module 41, on which the RPC server is deployed, after which the module 41 transmits the received request to the log management module 44 for registering it in a log located, for example, in the data storage module 45. For example, to register a request, the module 44 may extract the function ID and its parameters from said request and save them in the module 45.

[0043] The said request is also sent by module 41 to component simulation module 42, which, based on the stored information about the state of the simulated VRs, VR parameters contained in the request (for example, the function ID and the amount of required resources), generates a response to the request and, if necessary, updates the state of the simulated VRs. The initial state of the simulated VRs is set by module 43 at the time of initialization of module 42 based on the data received from the user. In a particular embodiment of the presented solution, the list of states of the simulated VRs, VR parameters and the responses corresponding to them can be pre-set in the memory of module 42 by the developer. In another embodiment, the developer implements an algorithm in module 42 that allows constructing a response based on a set of states of the simulated VRs and parameters contained in the request.

[0044] For example, the information about the state of the simulated VR may characterize the available memory capacity of the video card in accordance with the VR configuration, and the VR parameters contained in the request may indicate that it is necessary to allocate a specified amount of video card memory. The module 42 compares the available memory capacity with the required memory value, and if the required memory value does not exceed the available memory capacity, then the module 42 updates the information about the available memory capacity by increasing the required memory value by the amount.

[0045] Information about the updated state of the simulated VRs and the generated response to the request are sent to module 44 for storage in module 45. Also, the generated response to the request and the indicator of successful execution of the request are sent to module 41, which returns the received data to server 30 in the isolated environment and completes the remote RPC call.

[0046] This ensures that the program code of the application is executed on a server that is not equipped with the computing resources required to run the said application. In the case of using other resources, such as RAM or the computing resources of the processor, the resources of the server on which the application is running are used. After the execution of the user program is completed, the results of the execution of the user program are transferred to the user (for example, the text intended for user program for display on the screen), a log of calls to functions of replacement libraries (collected by storage module 45) and resource usage statistics (contained in storage module 45).

[0047] As a result of using the proposed solution, it was possible to: - start training the GPT3 neural network using the Megatron LM framework; - detect several errors in the source code that prevented successful learning; - select parallelism parameters for launching on 3072 video cards - compare different network topologies and select optimal parameters for each of them.

[0048] To test the prototype of the proposed invention, an application was used to perform the GPT3 training task using the MegatronLM framework for one epoch. The prototype successfully completed the task of executing the program code of the application. Executing the code of this application on a real cluster would require at least 1024 video card hours and 8192 core hours on the CPU (using Nvidia T4 and Intel 6226R). Executing the code of this application using the prototype of the invention required 850 core hours on the Intel 6226R CPU and did not require the use of video cards.

[0049] In addition to testing large tasks (hundreds of GPUs), the project can be used by small teams to test applications for several GPUs. For such teams, renting even several GPUs for testing represents a significant expense item. In general (see Fig. 2), the computing device contains one or more processors (201) united by a common information exchange bus, memory means such as RAM (202) and ROM (203), input / output interfaces (204), input / output devices (205), and a device for network interaction (206).

[0050] The processor (201) (or several processors, a multi-core processor, etc.) can be selected from a range of devices that are widely used at present, for example, from such manufacturers as: Intel™, AMD™, Apple™, Samsung Exynos™, MediaTEK™, Qualcomm Snapdragon™, etc. The processor or one of the processors used in the device (200) must also include a graphics processor, for example, an NVIDIA or Graphcore GPU, the type of which is also suitable for the full or partial implementation of the method, and can also be used for training and applying machine learning models in various information systems.

[0051] RAM (202) is a random access memory and is intended for storing machine-readable instructions executed by the processor (201) to perform the necessary operations for logical data processing. RAM (202), as a rule, contains executable instructions of the operating system and the corresponding software components (applications, software modules, etc.). In this case, the available memory capacity of the graphic card or graphic processor can act as RAM (202).

[0052] ROM (203) is one or more permanent data storage devices, such as a hard disk drive (HDD), a solid-state drive (SSD), flash memory (EEPROM, NAND, etc.), optical storage media (CD-R / RW, DVD-R / RW, BlueRay Disc, MD), etc.

[0053] To organize the operation of the device components (200) and to organize the operation of external connected devices, various types of I / O interfaces (204) are used. The choice of the corresponding interfaces depends on the specific design of the computing device, which may include, but are not limited to: PCI, AGP, PS / 2, IrDa, FireWire, LPT, COM, SATA, IDE, Lightning, USB (2.0, 3.0, 3.1, micro, mini, type C), TRS / Audio jack (2.5, 3.5, 6.35), HDMI, DVI, VGA, Display Port, RJ45, RS232, etc.

[0054] To ensure interaction between the user and the device (200), various means (205) of I / O information are used, for example, a keyboard, a display (monitor), a touch display, a touchpad, a joystick, a mouse, a light pen, a stylus, a touch panel, a trackball, speakers, a microphone, augmented reality means, optical sensors, a tablet, light indicators, a projector, a camera, biometric identification means (a retina scanner, a fingerprint scanner, a voice recognition module), etc.

[0055] The network interaction means (206) ensures data transmission via an internal or external computer network, such as Intranet, Internet, LAN, etc. One or more means (206) may be, but are not limited to: an Ethernet card, GSM modem, GPRS modem, LTE modem, 5G modem, satellite communication module, NFC module, Bluetooth and / or BLE module, Wi-Fi module, etc.

[0056] The specific selection of device elements (200) for the implementation of various software and hardware architectural solutions may vary while maintaining the required functionality provided.

[0057] Modifications and improvements of the above-described embodiments of the present technical solution will be clear to those skilled in the art. The preceding description is provided only as an example and does not bear any limitations. Thus, the scope of the present technical solution is limited only by the scope of the attached claims.

Claims

CLAUSE OF INVENTION.

1. A method for running an application in a simulated environment, performed by means of at least one computing device, comprising the steps of: - receive a request to launch an application, containing the configurations of computing resources (CR) required to launch the application; - based on the VR configuration, components simulating VR are deployed; - based on the VR configuration, at least one simulated server is deployed, containing replacement libraries intended for transmitting commands addressed to the VR to the simulated VRs, with each simulated server corresponding to an isolated environment; - place the VR configurations and the application in at least one isolated environment; - execute the program code in at least one isolated environment in accordance with the VR configuration; - using replacement libraries, redirect requests addressed to VR to components simulating VR; - generate responses to the mentioned requests addressed to the VR by processing them by components simulating the VR; - return the generated responses to the said requests to at least one isolated environment; - terminate the execution of program code in an isolated environment.

2. The method according to item 1, characterized in that the VR configurations contain: - description of video card models, presented as a set of parameters; - description of network card models, presented as a set of parameters; - information about the number of simulated servers and a set of parameters for each simulated server; - network connection parameters; - application to launch.

3. The method according to claim 1, characterized in that the execution of the program code is carried out in at least two isolated environments, and the transfer of data from an instance of the application executed in the first isolated environment to an instance of the application executed in the second isolated environment is carried out via a virtual network, if for said transfer data requires the use of network resources using the functionality of the operating system.

4. The method according to item 1, characterized in that the requests addressed to the BP contain at least one of: BP parameters characterizing the BP ID, the function ID, the number of required resources.

5. The method according to item 1, characterized in that the components simulating the VR generate responses to the said requests based on stored information about the state of the simulated VR and the VR parameters contained in the request.

6. The method according to paragraph 1, characterized in that the received requests and the state of the simulated VRs are recorded in a log.

7. The method according to item 1, characterized in that the step of updating the state of the simulated VR is additionally performed after generating a response to the mentioned requests addressed to the VR.

8. The method according to item 1, characterized in that the following steps are additionally performed: - deploy an RPC server; - place the address of the RPC server in at least one isolated environment; wherein said redirection of requests and generated responses is performed using the RPC server.

9. A system for launching applications in a simulated environment, comprising at least one computing device and at least one memory containing machine-readable instructions that, when executed by at least one computing device, perform the method according to any of paragraphs 1-8.

Images