System and migration method

Abstract

To provide a technique for migrating virtual machines and GPUs in conjunction with each other.SOLUTION: A system includes: a first device in which a virtual machine operates; a second device that is a migration destination of the virtual machine; a third device that is connected to the first device and on which a processor allocated to the virtual machine is mounted; and means for migrating the virtual machine from the first device to the second device. When migrating the virtual machine from the first device to the second device, the means for migration separates the third device from the first device, connects the third device to the second device, and allocates the processor to the virtual machine migrated to the second device, thereby starting the virtual machine.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a system and a migration method. [Background technology]

[0002] With the release of ChatGPT (registered trademark) by OpenAI (registered trademark), large-scale language models (LLMs) with 10-100B parameters have attracted attention. OpenAI (registered trademark)'s GPT-3 (registered trademark) is said to have 175B parameters, and GPT-3.5 has 355B parameters. Training an LLM with a large number of parameters requires long training times using multiple graphics processing units (GPUs). In reality, downtime due to maintenance and failures during training must also be taken into consideration. Failures that occur include hardware failures such as power outages and disk failures, operating system (OS) failures such as downtimes and stalls, and software failures. When a server fails, training is suspended until recovery. To complete training in a shorter time, methods for stabilizing the system and reducing failure recovery times are needed.

[0003] A conventional technique for stabilizing systems and reducing recovery time is the migration of virtual machines (VMs) using clustering software. Clustering software controls a cluster configuration consisting of one active server and one or more backup servers. Figure 1 shows an example system configuration. In the example shown in Figure 1, active server 1 and backup server 2 are connected to a shared disk 4 via network interface cards (NICs) 3a and 3b. Active server 1 is equipped with a CPU 5a and a GPU 7a. CPU 5a runs OS 6a, which is equipped with clustering software 8a and a hypervisor 9a, which creates and runs virtual machines. Backup server 2 is similarly equipped with a CPU 5b and a GPU 7b, which runs OS 6b, which is equipped with clustering software 8b and a hypervisor 9b. In addition, a Virtual Machine 10 is constructed on a Hypervisor 9a in the active server 1, and a GPU Driver 11 that executes the initialization process of the GPU 7a and an LLM Application 12 for executing large-scale learning are implemented in the Virtual Machine 10.

[0004] If a failure occurs on the active server 1, the virtual machine 10 can be transferred to the backup server 2 via the shared disk 4, as shown in Figure 2. When the virtual machine 10 migrated to the backup server 2 is started, the GPU driver 11 executes an initialization process, resetting the memory of the GPU 7b and erasing the GPU data from the training on the active server 1. However, since the LLM application 12 can save the intermediate training results as a checkpoint file on the shared disk 4 when the LLM application 12 is started on the backup server 2, the checkpoint file can be retrieved from the shared disk 4 and the data can be loaded into the GPU 7b. Even if the training on the active server 1 is interrupted, the training can be resumed from the checkpoint on the backup server 2. However, simply migrating the virtual machine 10 requires GPUs on both the active server 1 and the backup server 2. Therefore, the GPU 7b installed on the backup server 2 must wait until a failure occurs on the active server 1, which reduces the GPU utilization rate relative to the total number of GPUs in the data center. Furthermore, the standby GPU7b consumes power, increasing costs and carbon dioxide emissions.

[0005] Furthermore, when the Virtual Machine 10 is started after being migrated to the backup server 2, the initialization process of the GPU 7b is executed and the memory is reset. Therefore, the data used during training on the active server 1 must be loaded from the shared disk 4 to the GPU 7b, and the migration takes time.

[0006] One way to reduce GPU costs is to share GPU-equipped devices using MR-IOV (Multi-Root I / O Virtualization). MR-IOV is a standard that uses a dedicated PCIe (Peripheral Component Interconnect Express) switch to enable sharing of PCIe devices among multiple operating systems. However, dedicated PCIe switches are expensive and lack versatility. One possible method using an inexpensive Ethernet switch is GPU migration using PCIe over Ethernet technology. PCIe over Ethernet technology enables the expansion of PCIe interconnects, which are used to connect various PCIe devices within a server, to a general-purpose network such as Ethernet. Figure 3 shows an example system configuration. Server 14 is equipped with a CPU 17a and a PCIe Expander Engine (Server) (hereinafter referred to as EE (Server)) 18, which has a non-volatile memory (NVM) 22a. Similarly, server 15 is equipped with a CPU 17b and an EE (Server) 19, which has a NVM 22b. The I / O expansion enclosure 16 is equipped with a GPU 21 and a PCIe Expander Engine (IO) (hereinafter referred to as EE(IO)) 20, and the EE(IO) 20 has an NVM 22c. EEs (Servers) 18 and 19 have server-side functions, and the EE(IO) 20 has I / O-side functions. The servers 14 and 15 and the I / O expansion enclosure 16 are connected to a general-purpose network 13. The EEs (Servers) 18 and 19 and the EE(IO) 20 communicate with each other via the general-purpose network 13. When transferring PCIe packets over Ethernet, the PCIe packets are encapsulated in Ethernet frames and transferred. The NVMs 22a to 22c hold Group IDs and connect between EEs (Servers) and EEs (IOs) that have the same Group ID.3, NVM 22a and NVM 22c have Group ID=1, and NVM 22b has Group ID=2, so EE (Server) 18 and EE (IO) 20 are connected. When Ethernet is used for the general-purpose network 3, this can be configured using a VLAN setting. For example, by changing the Group ID of EE (IO) 20 from Group ID=1 of EE (Server) 18 of the running server to Group ID=2 of EE (Server) 19 of the migration destination server, the connection destination of I / O expansion enclosure 16 can be changed from server 14 to server 15, as shown in FIG. 4.

[0007] For example, suppose a virtual machine running on server 14 is assigned a GPU 21 in an I / O expansion enclosure 16. If a failure occurs on server 14 while the virtual machine's application is running large-scale LLM training, the virtual machine and GPU must be migrated in tandem to continue the large-scale LLM training. It is conceivable that the GPU can be migrated by migrating the virtual machine from server 14 to server 15 and changing the Group ID of the EE(IO) 20 from 1 to 2. However, virtual machine migration using clustering software is a software-based technology, while PCIe over Ethernet technology is a hardware-based technology. To integrate these technologies from different domains into a system, it is necessary to monitor and coordinate the virtual machine and devices. More specifically, to shorten the total training time by taking into account potential maintenance and failures during large-scale LLM training, a mechanism is needed to coordinate the migration using clustering software with the switching of the I / O expansion enclosure 16 connected via PCIe over Ethernet. However, such a mechanism has not yet been established.

[0008] For example, Patent Document 1 discloses a system that provides a network service, and automatically migrates the virtual machine that realizes the function of the service to another piece of hardware, thereby enabling the network service to continue even when a hardware failure occurs or maintenance is performed. However, this technology does not link migration by clustering software with switching of devices connected via a network (I / O expansion enclosure 16 in the above example). [Prior art documents] [Patent documents]

[0009] [Patent Document 1] International Publication No. 2022 / 172063 Summary of the Invention [Problem to be solved by the invention]

[0010] One of the objects is to provide a method for linking the migration of a virtual machine with the process of switching the connection destination of a device connected to the virtual machine via a network to the virtual machine after migration. [Means for solving the problem]

[0011] According to one aspect of the present disclosure, a system includes a first device on which a virtual machine runs, a second device to which the virtual machine is migrated, a third device connected to the first device and equipped with a processor assigned to the virtual machine, and a means for migrating the virtual machine from the first device to the second device, wherein when migrating the virtual machine from the first device to the second device, the migration means switches the connection destination of the third device from the first device to the second device, and assigns and starts the processor to the virtual machine migrated to the second device.

[0012] According to one aspect of the present disclosure, a migration method is provided in a system including a first device on which a virtual machine runs, a second device to which the virtual machine is migrated, and a third device connected to the first device and equipped with a processor assigned to the virtual machine, wherein when the virtual machine is migrated from the first device to the second device, the connection destination of the third device is switched from the first device to the second device, and the processor is assigned to and started up the virtual machine migrated to the second device. [Effects of the Invention]

[0013] According to the present disclosure, it is possible to link the migration of a virtual machine with the process of switching the connection destination of a device connected to the virtual machine via a network to the migrated virtual machine. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a first diagram illustrating a general migration of a virtual machine. [Figure 2] FIG. 2 is a second diagram illustrating a general migration of a virtual machine. [Figure 3] FIG. 1 is a first diagram illustrating switching of a connected device in a system including devices connected via PCIe over Ethernet. [Figure 4] FIG. 2 is a second diagram illustrating switching of a connected device in a system including devices connected via PCIe over Ethernet. [Figure 5] FIG. 1 is a first diagram illustrating an example of a system configuration according to an embodiment. [Figure 6] FIG. 2 is a second diagram illustrating an example of a system configuration according to an embodiment. [Figure 7] 10 is a flowchart illustrating an example of a migration process according to the embodiment. [Figure 8]10 is a flowchart illustrating an example of initialization processing of a GPU driver according to the embodiment. [Figure 9] 10 is a flowchart illustrating an example of processing when an application is started according to the embodiment. [Figure 10] FIG. 3 is a third diagram illustrating an example of a system configuration according to an embodiment. [Figure 11] 1 is a flowchart illustrating an example of a migration method according to an embodiment. [Figure 12] FIG. 1 is a diagram illustrating an example of a hardware configuration of a system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Systems according to embodiments of the present disclosure will be described below with reference to the drawings. In the drawings used in the following description, components that are not related to the present disclosure may be omitted or not shown. The same or equivalent components are designated by the same reference numerals in all drawings, and common descriptions may be omitted.

[0016] First Embodiment (System Configuration) 5 and 6 show a system 100 according to an embodiment. In the system 100, an active server 24, a backup server 25, an I / O expansion enclosure 26, a device control server 27, and a shared disk 42 are connected to a general-purpose network 23. In addition, one or more backup servers (not shown) may be connected to the general-purpose network 23. The active server 24 is equipped with a CPU 28a, an EE (Server) 29, and a NIC 44a. The CPU 28a runs an OS 36a, and clustering software 37a and a hypervisor 38a are implemented in the OS 36a. The EE (Server) 29 is equipped with an NVM 33a and a MAC 34a. A virtual machine 39 is constructed in the hypervisor 38a, and a GPU driver 40 and an LLM application 41 are implemented in the virtual machine 39. The backup server 25 is equipped with a CPU 28b, an EE (Server) 30, and a NIC 44b. The CPU 28b runs an OS 36b, which is equipped with clustering software 37b and a hypervisor 38b. The EE (Server) 30 includes an NVM 33b and a MAC 34b. The I / O expansion enclosure 26 includes an EE (IO) 31 and a GPU 32, and the EE (IO) 31 includes an NVM 33c and a MAC 34c. The EE (Server) 29-30 are hardware and software modules equipped with server-side functions for realizing communication connections using PCIe over Ethernet technology, and the EE (IO) 31 is a hardware and software module equipped with IO device-side functions for realizing communication connections using PCIe over Ethernet technology. The NVMs 33a-33c are non-volatile memories. The MACs 34a-34c are devices at the MAC layer of the network, each assigned a MAC address. The EE (Server) 29-30 and the EE (IO) 31 communicate with each other via the general-purpose network 23. The device control server 27 includes a public manager 35.The public manager 35 can change the connection destination of the IO expansion enclosure 26 by rewriting the Group ID. The public manager 35 also exchanges commands with the clustering software 37a and 37b, and migrates the virtual machine 39 and the GPU 32 in conjunction with each other. A virtual hard disk image 43 for starting the virtual machine 39 is stored in the shared disk 42. The shared disk 42 is connected to the CPU 28a via a NIC 44a. The shared disk 42 is connected to the CPU 28b via a NIC 44b.

[0017] (operation) Next, the operation of the system 100 will be described. FIG. 7 is a flowchart showing an example of migration processing according to the embodiment. Assume the configuration illustrated in FIG. 5. First, the hypervisor 38a of the active server 24 starts the virtual machine 39 using the virtual hard disk image 43 stored on the shared disk 42 (step S201). Next, the LLM application 41 is started on the started virtual machine 39, and large-scale learning begins (step S202). The application 41 saves intermediate results of the learning as checkpoints in a file on the shared disk 42 as needed (for example, at predetermined intervals or whenever a predetermined process is completed) (step S203). If a failure occurs on the active server 24, the clustering software 37a detects the failure (step S204). Upon detecting the failure, the clustering software 37a executes a script to stop the virtual machine 39 of the hypervisor 38a (step S205). The clustering software 37a then selects the destination backup server 25 from the group of multiple backup servers connected to the general-purpose network 23, and acquires information about the backup server 25 (e.g., identification information, IP address, MAC address, etc.) (step S206). After acquiring the information, the clustering software 37a notifies the device control server 27 to start the public manager 35 (step S207). The device control server 27 starts the public manager 35 in response to the notification from the clustering software 37a of the active server 24 (step S208).

[0018] Next, a process for switching the GPU 32 is executed. First, the public manager 35 acquires the address of the MAC 34b of the EE (Server) 30 of the backup server 25 based on the information acquired by the clustering software 37a of the active server 24. Then, the public manager 35 uses the acquired address to access the NVM 33b of the EE (Server) 30 of the backup server 25 and acquires the Group ID (step S209). Next, the public manager 35 changes the value of the Group ID stored in the NVM 33c of the EE (IO) 31 from the value of the Group ID stored in the NVM 33a of the EE (Server) 29 of the active server 24 to the value of the Group ID stored in the NVM 33b of the EE (Server) 30 of the backup server 25. This changes the connection destination of the I / O expansion enclosure 26 to the backup server 25 by hot plugging (step S210). At this time, the public manager 35 sets a flag in the NVM 33c of the EE (IO) 31 indicating that the GPU 32 is a migration target. After the change of the connection destination is completed, the public manager 35 notifies the clustering software 37b of the backup server 25 to allocate the GPU 32 to the Virtual Machine 39 (step S211). The clustering software 37b of the backup server 25 starts operation upon receiving the notification from the public manager 35.

[0019] Next, the processing procedure in the backup server 25 will be described with reference to the configuration illustrated in FIG. 6. First, the GPU 32 is assigned to the virtual machine 39 (step S212). The clustering software 37b assigns the GPU 32 to the virtual machine 39. Next, the hypervisor 38b of the backup server 25 starts the virtual machine 39 using the virtual hard disk image 43 stored in the shared disk 42 (step S213). When the virtual machine 39 starts, an initialization process of the GPU driver 40 is executed (step S214). Then, the LLM application 41 is started in the virtual machine 39 of the backup server 25 (step S215). Finally, the LLM application 41 resumes learning of the LLM (step S216). After learning is resumed, the flag recorded in the NVM 33c indicating that the GPU 32 is a migration target is initialized by the public manager 35.

[0020] The detailed operation of the initialization process of the GPU driver 40 (step S214 in FIG. 7) will be described with reference to FIG. 8. The process in FIG. 8 is executed by the hypervisor 38b and the GPU driver 40 in cooperation with each other. First, when the initialization process is started, the NVM 33c of the EE(IO) 31 is accessed to check whether a flag indicating that the GPU 32 is a migrated GPU is set (step S301). If the flag is set (step S301; Yes), execution of a memory reset command for the GPU 32 is stopped during the initialization process of the GPU Driver 40 (step S302). This prevents the memory reset of the GPU 32. If the flag is not set (step S301; No), the GPU is not a migrated GPU, so the memory reset command for the GPU 32 is executed. This resets the memory of the GPU 32 (step S303).

[0021] Next, detailed operations at the time of startup of the LLM Application 41 (step S215) will be described with reference to Fig. 9. The processing in Fig. 9 is executed by the Hypervisor 38b and the GPU Driver 40 in cooperation with each other. First, the NVM 33c of the EE(IO) 31 is accessed to check whether a flag indicating that the GPU is a migrated GPU is set (step S304). If the flag is set (step S304; Yes), the memory of the GPU 32 has not been reset and data from the learning on the active server 24 remains, so no data is loaded into the memory of the GPU 32. This reduces the data loading time while allowing the intermediate results of the learning on the active server 24 to be inherited (step S305). If the flag is not set (step S304; No), the memory of the GPU 32 was reset when the GPU initialization process was executed, so data saved as a checkpoint is obtained from the shared disk 42 and loaded into the memory of the GPU 32 (step S306).

[0022] (effect) As described above, according to the first embodiment, the migration of the virtual machine 39 using the clustering software 37a and 37b and the migration of the GPU 32 using PCIe over Ethernet technology (EE (Server) 29, 30 and EE (IO) 31) are linked as a system, thereby migrating the virtual machine and the GPU in conjunction with each other. This eliminates the need to prepare a GPU for the backup server, and the GPU for the active server can be installed on the backup server, thereby improving the GPU utilization rate. By switching the GPU for the active server to the backup server, the number of standby GPUs can be reduced, thereby realizing reductions in power consumption and carbon dioxide emissions throughout the data center. Furthermore, since the backup server can inherit the results of training on the active server without loading GPU data, the time required for migration can be reduced. Furthermore, since this can be achieved using a general-purpose Ethernet switch, the system can be built inexpensively.

[0023] 5 and 6, it is also possible to use a case where there is not a single backup server 25 but multiple backup servers 25 connected to the general-purpose network 23. The backup server to which the virtual machine 39 or GPU 32 is transferred can be specified by the clustering software 37a. It is also possible to use a case where there is not a single I / O expansion enclosure 26 but multiple I / O expansion enclosures 26 connected to the general-purpose network 23. The GPU to be migrated can be identified by specifying the address of the MAC 34 of the EE (IO) to the public manager 35.

[0024] In the above embodiment, when migrating a virtual machine 29 that performs large-scale learning using a GPU 32, the virtual machine 29 and the GPU 32 are migrated in cooperation with each other. However, the IO device that is linked with the virtual machine 29 is not limited to a GPU. For example, even if the GPU 32 is replaced with a NIC, storage, USB board, capture board, or the like mounted in the I / O expansion housing 26, the virtual machine 29 can be migrated in cooperation with these IO devices in the same manner as in the above embodiment.

[0025] Second Embodiment FIG. 10 is a third diagram illustrating an example of a system configuration according to the embodiment. The system 806 comprises a first device 800 on which a virtual machine 801 runs, a second device 802 to which the virtual machine is migrated, a third device 803 connected to the first device 800 and equipped with a processor 804 assigned to the virtual machine 801, and a migration means 805 for migrating the virtual machine 801 from the first device 800 to the second device 802, wherein when the migration means 805 migrates the virtual machine 801 from the first device 800 to the second device 802, the migration means 805 disconnects the third device 803 from the first device 800 and connects it to the second device 802, and assigns the processor 804 to the virtual machine 801 migrated to the second device 802 and starts it up.

[0026] FIG. 11 is a flowchart showing an example of a migration method according to the embodiment. In a system 806 including a first device 800 running a virtual machine 801, a second device 802 to which the virtual machine 801 is migrated, and a third device 803 connected to the first device 800 and equipped with a processor 804 allocated to the virtual machine 801, when the virtual machine 801 is migrated from the first device 800 to the second device 802, the connection destination of the third device 803 is switched (step S801). The third device 803 is disconnected from the first device 800 and connected to the second device 802. Next, the processor is allocated to the virtual machine 801 migrated to the second device 802 (step S802). Next, the virtual machine 801 of the second device 802 is started (step S803).

[0027] FIG. 12 is a diagram illustrating an example of a hardware configuration of a system according to an embodiment. The computer 900 includes a CPU 901, a main memory device 902, an auxiliary memory device 903, an input / output interface 904, and a communication interface 905. The above-described systems 100 and 806 are implemented in the computer 900. The above-described functions are stored in the auxiliary memory device 903 in the form of a program. The CPU 901 reads the program from the auxiliary memory device 903, loads it into the main memory device 902, and executes the above-described processing in accordance with the program. The CPU 901 also allocates a storage area in the main memory device 902 in accordance with the program. The CPU 901 also allocates a storage area in the auxiliary memory device 903 for storing data being processed in accordance with the program.

[0028] Alternatively, a program for implementing all or part of the functions of the systems 100 and 806 may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed to perform processing by each functional unit. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, the term "computer-readable recording medium" refers to portable media such as CDs, DVDs, and USBs, as well as storage devices such as hard disks built into the computer system. Furthermore, if the program is distributed to the computer 900 via a communication line, the computer 900 that receives the program may load the program into the main storage device 902 and execute the above-described processing. Furthermore, the program may be for implementing part of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system.

[0029] Although one embodiment of the present disclosure has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications are possible within the scope of the gist of the present invention. Furthermore, one aspect of the present disclosure may be modified in various ways within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. Furthermore, configurations in which elements described in the above embodiments and variations are substituted with elements that achieve the same effect are also included. Furthermore, each embodiment may be combined with other embodiments as appropriate.

[0030] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.

[0031] (Appendix 1) The system comprises a first device on which a virtual machine runs, a second device to which the virtual machine is migrated, a third device connected to the first device and equipped with a processor such as a GPU assigned to the virtual machine, and a means for migrating the virtual machine from the first device to the second device, wherein when migrating the virtual machine from the first device to the second device, the migration means disconnects the third device from the first device and connects it to the second device, and assigns the processor to the virtual machine migrated to the second device.

[0032] (Appendix 2) The system described in Appendix (1) is such that the first device and the third device are connected via PCIe over Ethernet, and the migration means, when migrating the virtual machine from the first device to the second device, disconnects the PCIe over Ethernet connection between the third device and the first device and connects the third device and the second device via PCIe over Ethernet.

[0033] (Appendix 3) The migration means is a system described in Appendix (2) that switches the connection destination of the third device to the second device by rewriting the connection destination information of the PCIe expander Engine of the third device from the first device to the second device.

[0034] (Appendix 4) The migration means is a system described in Supplementary Notes (1) to (3) that, when allocating and starting the processor to the virtual machine of the second device, disables resetting of the processor's memory and starts it up.

[0035] (Appendix 5) The migration means is a system described in Appendix (4) that has the virtual machine started on the second device execute the processing without loading information on the processing results processed by the virtual machine on the first device before migration into the memory of the processor.

[0036] (Appendix 6) The system is a system of Appendix (5) further comprising a storage device accessible from both the first device and the second device, wherein the virtual machine of the first device executes an LLM learning process, and information on the progress of the learning process is recorded in the storage device as a checkpoint, and when the first device becomes unusable during the learning process, the migration means continues the learning process on the second device without loading the information recorded in the storage device into the memory of the processor.

[0037] (Appendix 7) The migration means includes a hypervisor, clustering software, and connection means of the first device, a hypervisor, clustering software, and connection means of the second device, a connection means of the third device, and a means for controlling the connection means of the first device, the connection means of the second device, and the connection means of the third device, and when migrating the virtual machine from the first device to the second device, the clustering software of the first device instructs the hypervisor of the first device to stop the virtual machine and instructs the connection destination switching means to switch the connection destination of the processor, and the connection destination switching means changes the connection destination of the third device to the connection destination of the first device. and switches the processor from the first device to the second device, records a flag in the connection means of the third device indicating that the processor is a target for migration, and instructs the clustering software of the second device to assign the processor to the virtual machine, the clustering software of the second device assigns the processor to the virtual machine, and a hypervisor of the second device starts the virtual machine, and when the virtual machine is started, the hypervisor of the second device checks whether the flag is present when accessing the processor through the connection means of the third device, and if the flag is present, does not reset the memory of the processor.

[0038] (Appendix 7-1) The system described in Appendix (6) is such that, when the virtual machine is started, the hypervisor of the second device checks whether the flag is present when accessing the processor through the connection means of the third device, and if the flag is not present, resets the memory of the processor.

[0039] (Appendix 8) The system described in appendix (1) to appendix (7) is one in which the migration means migrates the virtual machine and IO devices other than the processor mounted on the third device from the first device to the second device.

[0040] (Appendix 9) The system according to claim 8, wherein the IO device is one of a NIC, a storage, a USB board, and a capture board.

[0041] (Appendix 10) In a system including a first device on which a virtual machine runs, a second device to which the virtual machine is migrated, and a third device connected to the first device and equipped with a processor such as a GPU assigned to the virtual machine, when migrating the virtual machine from the first device to the second device, a migration method is provided in which the third device is disconnected from the first device and connected to the second device, and the processor is assigned to and started up by the virtual machine migrated to the second device. [Explanation of symbols]

[0042] 1. Active Server 2. Backup server 3a, 3b...NIC 4. Shared disk 5a, 5b...CPU 6a, 6b...OS 7a, 7b... GPU 8a, 8b Clustering software 9a, 9b... Hypervisor 10. Virtual Machine 11. GPU Driver 12. Application 13 General-purpose network 14. Server 15. Server 16 I / O expansion cabinet 17a, 17b CPU 18, 19 EE (Server) 20···EE(IO) 21. GPU 22a, 22b, 22c NVM 23 General-purpose network 24 Active Servers 25 Backup Server 26 I / O expansion cabinet 27. Device control server 28a, 28b...CPU 29, 30 EE (Server) 31···EE(IO) 32 GPUs 33a, 33b, 33c NVM 34a, 34b, 34c...MAC 35. Public Manager 36a, 36b···OS 37a, 37b Clustering software 38a, 38b... Hypervisor 39. Virtual Machine 40···GPU Driver 41···Application 44a, 44b...NIC 42 Shared Disk 43 Virtual Hard Disk Image 100···System 900···Computer 901 CPU 902...Main memory 903...Auxiliary storage device 904 Input / Output Interface 905···Communication Interface

Claims

1. a first device on which a virtual machine runs; a second device to which the virtual machine is migrated; a third device connected to the first device and including a processor assigned to the virtual machine; means for migrating the virtual machine from the first device to the second device; Equipped with the migration means, when migrating the virtual machine from the first device to the second device, switches the connection destination of the third device from the first device to the second device, and assigns the processor to the virtual machine migrated to the second device and starts it up; system.

2. the first device and the third device are connected via PCIe over Ethernet; the migration means, when migrating the virtual machine from the first device to the second device, disconnects the connection between the third device and the first device via PCIe over Ethernet and connects the third device and the second device via PCIe over Ethernet; The system of claim 1 .

3. The migration means includes: rewriting connection destination information of a PCIe Expander Engine of the third device from the first device to the second device, thereby switching the connection destination of the third device to the second device; The system of claim 2 .

4. The migration means includes: When the processor is assigned to the virtual machine of the second device and the virtual machine is started, reset of the memory of the processor is disabled and the virtual machine is started.

3. The system according to claim 1 or claim 2.

5. The migration means includes: causing the virtual machine started on the second device to execute the processing without loading information on the processing result of the processing performed by the virtual machine on the first device before migration into a memory of the processor; The system of claim 4.

6. the system further comprises a storage device accessible from both the first device and the second device; In the virtual machine of the first device, a learning process of an LLM is executed, and result information of the learning process is recorded in the storage device at each checkpoint; If the first device becomes unavailable during the learning process, the migration means continues the learning process on the second device without loading the result information recorded in the storage device into the memory of the processor. The system of claim 5.

7. The migration means includes: a hypervisor, clustering software and connection means of said first device; a hypervisor, clustering software and connection means of said second device; connection means of the third device; a means for controlling the connection means of the first device, the connection means of the second device, and the connection means of the third device; When migrating the virtual machine from the first device to the second device, the clustering software of the first device instructs the hypervisor of the first device to stop the virtual machine and instructs the control means to switch the connection destination of the processor; the control means switches the connection destination of the third device from the first device to the second device, records a flag indicating that the processor is a migration target in the connection means of the third device, and instructs the clustering software of the second device to assign the processor to the virtual machine; The clustering software of the second device assigns the processors to the virtual machines; a hypervisor on the second device launches the virtual machine; When starting up the virtual machine, the hypervisor of the second device checks whether or not the flag is present when accessing the processor through the connection means of the third device, and if the flag is present, does not reset the memory of the processor. The system of claim 1 .

8. The migration means includes: Migrating the virtual machine and an IO device other than the processor mounted on the third device from the first device to the second device; 3. The system according to claim 1 or claim 2.

9. The IO device is any one of a NIC, a storage, a USB board, and a capture board. The system of claim 8.

10. a first device on which a virtual machine runs; a second device to which the virtual machine is migrated; a third device connected to the first device and including a processor assigned to the virtual machine, When migrating the virtual machine from the first device to the second device, switching the connection destination of the third device from the first device to the second device, and allocating the processor to the virtual machine migrated to the second device and starting it up. Migration methods.

Images