A multi-virtual machine hot migration data deduplication method in a cloud computing environment

Abstract

This invention discloses a method for deduplicating multi-virtual machine hot migration data in a cloud computing environment, belonging to the field of cloud computing technology. The specific steps include: continuously learning from virtual machines in a test environment to master a certain level of virtual machine memory data hash digest->value pairs based on the operating system image ID, and understanding the data hit rate; and importing the data learned from the test environment into a migration cache server to complete the deduplication of multi-virtual machine hot migration data. This invention can effectively reduce virtual machine hot migration time, while reducing the pressure of virtual machine migration on the business network and mitigating business risks for other cloud hosts.

Description

A method for deduplication of multi-virtual machine hot migration data in a cloud computing environment Technical Field

[0001] This invention belongs to the field of cloud computing technology, specifically a method for deduplication of multi-virtual machine hot migration data in a cloud computing environment. Background Technology

[0002] In large-scale cloud computing environments, mass live migration of virtual machines (VMs) is a frequent occurrence. Mass data migration often occurs due to hardware failures or software bugs on host machines rendering them unusable, necessitating the migration of VMs. Maintenance operations may also require firmware upgrades or software updates on certain host machines, all of which necessitate mass VM migration. Live VM migration involves copying a portion of the VM's data from the host machine's memory to the destination. For the tens of thousands of VMs in a cloud computing scenario, the image files loaded during creation are typically dozens of operating system images stored on the cloud platform. This means that a large number of VMs in the cloud platform environment are created based on the same image, and a significant portion of their memory data is identical. Therefore, during mass live VM migration, a large amount of duplicate memory data is copied, resulting in unnecessary network bandwidth consumption, impacting the operation of normal hosts, and increasing the overall migration time.

[0003] This invention proposes a method for deduplication of multi-virtual machine hot migration data in a cloud computing environment. Based on various operating system images on the cloud platform, the method extracts duplicate content in the memory data of virtual machines created on the same image through repeated hot migration, and performs real-time deduplication of the migration data in subsequent hot migration operations. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a method for deduplication of multi-virtual machine hot migration data in a cloud computing environment, which can effectively solve the problems in the background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solution, comprising the following specific steps:

[0006] S1. Through continuous learning of virtual machines in the test environment, master a certain level of virtual machine memory data hash digest->value pairs based on operating system image ID, and master the hit rate of its data;

[0007] S2. By importing the data learned in the test environment into the migration cache server, the deduplication of multi-virtual machine hot migration data is completed.

[0008] A further improvement of the present invention is that the specific step S1 includes:

[0009] S11. In the test environment, based on the operating system image provided on the cloud platform, a large number of virtual machines are created using a single image, and the corresponding test business models are configured.

[0010] S12. Perform repeated hot migration tests on the virtual machine. While the virtual machine is being migrated and its memory is copied, libvirtd uses a hash algorithm to calculate the hash value of each memory page in 4KB units. The hash value and the corresponding data in the memory are then temporarily stored in the memory database on the host client side.

[0011] S13. The client sends the memory page and its hash digest value to the migration cache server over the network. The server compares the hash digest value and checks whether the hash key already exists in the database. If it exists, the hit value of the corresponding block is incremented by 1; if it does not exist, the hash digest and page data are stored.

[0012] S14. By repeatedly migrating in the test environment and continuously updating the data values ​​on the server side, data with a hit rate of over 95% is extracted.

[0013] A further improvement of the present invention is that the hot migration process includes setting up a host A to be migrated and a host B to be migrated, and creating a set of virtual machines using n identical open-source operating systems on the host A to be migrated. vm n This refers to the nth virtual machine. At the start of the migration, the management platform sends a migration request for VM1 to host A (the host to be migrated), and determines the destination server to be migrated as host B. After the hypervisor of host A receives the migration request, it first sends a query request to the client to check if the cache contains the hash digest and memory page data for the operating system. If the client's cache does not contain the hash digest and memory page data for the operating system, it sends a request to the migration server to obtain the corresponding hash digest. The migration server returns a digest list value of the data with a hit rate of over 95% in the corresponding operating system to host A. Simultaneously, the hypervisor of host B also sends a query request to its client to check if the cache contains the hash digest and memory page data for the corresponding operating system. The migration server returns the digest and memory page data of the data with a hit rate of over 95% in the corresponding operating system to host B. At this point, the VM1 migration begins.

[0014] A further improvement of this invention is that, during the migration of VM1, memory data is copied. For the 4KB memory page that needs to be transferred, a hash digest value is calculated. Then, the local client is queried to see if the hash digest value exists. If the client has the hash digest value, it is sent to the hypervisor process of the migrated host B. If the client does not have the hash digest value, it means that the memory block does not have cached data, and the original migration method is followed, sending the corresponding memory data to the migrated host B. If the client does not have the hash digest value, the hash digest value and the content of the corresponding memory page are simultaneously transferred to the cache and sent to the server later. This allows each hot migration of the virtual machine to continuously learn from the cached data on the server.

[0015] A further improvement of this invention is that if the ype value on the host B to be migrated is a hash digest value, then the corresponding hash digest value is searched in the client cache to obtain the corresponding memory data, and the memory data copying process is completed. This process is repeated until the entire memory copying phase is over, and then the virtual machine state is switched from the host A to the host B to be migrated. Then, vm2 is migrated. After the vm2 migration is completed, the above process is repeated until the entire set of virtual machines is migrated.

[0016] A further improvement of the present invention is that the architecture of the client side includes: firstly, the client side will start an in-memory database Redis to cache the hash value of the in-memory page and the corresponding data; the backend is used to process the internal business logic; and the frontend provides APIs to communicate with the hypervisor and the server.

[0017] A further improvement of this invention is that the structure of the server-side includes: the server-side will run a MySQL database, and a table will be created based on each operating system on the cloud platform. Each table mainly stores five fields: hash, memory data, total number of migrations, number of migration hits, and migration hit rate; the backend is used to process internal business logic; the frontend provides APIs for communication with the hypervisor and client.

[0018] A further improvement of this invention is that the Hypervisor needs to modify the migration business logic. During the migration process, it first checks whether there is a hash digest for the memory page locally, sends pseudocode, and if the received business logic is a hash digest, it obtains the value of the corresponding digest through the client.

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method for deduplication of multi-virtual machine hot migration data in a cloud computing environment.

[0020] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for deduplication of multi-virtual machine hot migration data in a cloud computing environment.

[0021] Compared with the prior art, the beneficial effects of the present invention are:

[0022] 1. Current technologies for virtual machine hot migration in cloud environments primarily focus on optimizing the three modes of hot migration. The mainstream approach is to limit the CPU performance of virtual machines to reduce the generation of dirty pages in memory and shorten the virtual machine migration time. The technology of this invention innovatively addresses the situation in cloud computing environments where a large number of virtual machines are created based on a unified image. It proposes a method for deduplication of migration data in the case of batch virtual machine hot migration, which effectively reduces the virtual machine hot migration time, while reducing the pressure of virtual machine migration on the business network and reducing the business risks of other cloud hosts.

[0023] 2. Existing technologies cannot determine the data duplication rate during virtual machine migration in the overall cloud platform. This invention, while achieving data deduplication during virtual machine hot migration, can continuously learn the duplicated parts of virtual machine memory data based on various operating system images (parts with a hit rate higher than 95%) by refreshing the memory data hit rate.

[0024] 3. This invention can directly introduce duplicate virtual machine memory data into a newly created resource pool for deduplication operations by introducing the already known duplicate parts of virtual machine memory data. Attached Figure Description

[0025] Figure 1 shows a flowchart of a multi-virtual machine hot migration data deduplication method in a cloud computing environment according to the present invention. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0027] Example 1

[0028] This embodiment provides a method for deduplicating data during hot migration of multiple virtual machines in a cloud computing environment. It adopts a client-server model, consisting of a client deployed on the host for communication with the server and for caching data; a server is also required to store the hash digest:value of the memory data and its data hit rate; and the hypervisor needs to be equipped with corresponding functions.

[0029] Client-side: Deployed on the host machine, it mainly communicates with the server and the hepervisor, and also creates an in-memory database to cache page hash:value data with a duplicate hit rate of more than 95% in virtual machine memory.

[0030] Server-side: Deployed on the cloud platform, it communicates with the client and establishes a database to store the virtual machine memory data based on various system images, hash digest->value, and its data hit rate.

[0031] Hypervisor (KVM): Modified the connection establishment with the client, and added methods to calculate the memory page hash (MD5 / SHA1) value, query the client to see if the memory page hash value exists, and query the value corresponding to a specific hash value during the memory copying stage.

[0032] This invention first learns from virtual machines in the test environment to master a certain degree of virtual machine memory data hash digest->value pairs based on the operating system image ID, and understands the hit rate of its data;

[0033] Figure 1 illustrates the specific migration process of this invention. By importing the data learned in the test environment into the migration cache server, data deduplication for multi-virtual machine hot migration is completed. The hot migration process includes setting up a host A to be migrated and a host B to be migrated, and creating a set of virtual machines using n identical CentOS 7.6 images on the host A to be migrated. For example, VM n This refers to the nth virtual machine. At the start of the migration, the management platform sends a migration request for vm1 to the host A to be migrated, and determines that the destination server is host B. The specific steps include:

[0034] S201. When the migration begins, the management platform sends a migration request for VM1 to the host A to be migrated, and determines that the destination server to be migrated is the host B.

[0035] S202. After receiving the migration request, the hyperisor of the host A to be migrated first sends a query request to the client to check whether there is data of CentOS 7.6 hash digest and memory page in the cache.

[0036] S203. If the client's cache does not contain the hash digest and memory page data for CentOS 7.6, then send a request to the migration server to obtain the corresponding hash digest.

[0037] S204. The migration server returns the digest list values ​​of the corresponding CentOS 7.6 data with a hit rate of 95% or higher (which can be adjusted according to the host's memory usage) to the host A to be migrated.

[0038] S205. At the same time, the hypervisor of host B also sends a query request to the client of the migrated host B to check whether there is data of CentOS 7.6 hash digest and memory page in the cache.

[0039] S206. The migration server returns the digest and memory page data of the corresponding CentOS 7.6 data with a hit rate of over 95% to the migrated host B.

[0040] S207 and VM1 migration begins;

[0041] When migrating S208 and VM1, the system starts copying memory data. It needs to transfer 4KB of memory page data (the default memory page size in Linux is 4KB). First, it calculates the hash digest value and then checks whether the hash digest value exists on the local client.

[0042] S209. If the client has the hash digest value, send the hash digest value to the hypervisor process of the destination host B. If the client does not have the hash digest value, it means that the memory block does not have cached data. Then, follow the original migration method and send the corresponding memory data to the destination host B.

[0043] S210. If the client does not have the hash digest value, the hash digest value and the contents of the corresponding memory page are transferred to the cache and sent to the server later. This enables the server to continuously learn the cached data during each hot migration of the virtual machine.

[0044] S211. If the ype value on the host B being migrated is a hash digest value, then look up the corresponding hash digest value in the client cache, obtain the corresponding memory data, and complete the memory data copying process.

[0045] S212. Repeat this process until the entire memory copy phase is completed;

[0046] S213. Perform a virtual machine state switch, switching from host A to host B.

[0047] S214. Migrate VM2. Since the host A to be migrated and the host B to be migrated already have corresponding client cache data, the previous data preparation stage is no longer needed.

[0048] S215. The memory copying phase of the overall migration process shall be carried out in accordance with steps S208-S212.

[0049] The client-side architecture includes: firstly, the client starts an in-memory database, Redis, to cache the hash values ​​and corresponding data of in-memory pages; the backend is used to handle internal business logic; and the frontend provides APIs to communicate with the hypervisor and server.

[0050] The server-side structure includes: a MySQL database running on the server side, with one table created for each operating system on the cloud platform. Each table mainly stores five fields: hash, in-memory data, total number of migrations, number of migration hits, and migration hit rate; the backend is used to process internal business logic; and the frontend provides APIs for communication with the hypervisor and client.

[0051] The hypervisor needs to modify the migration business logic. During the migration process, it first checks if there is a hash digest of the memory page locally, and sends it as an example of pseudocode:

[0052] letpage[N]:=set of guest VM page N

[0053] function generate_digest(page){

[0054] Let algorithm="md5"##or let algorithm="sha256"

[0055] let digest_page=crypto.createHash(algorithm).update(page[N]).digest("hex")

[0056] return digest_page}

[0057] pageN_digest=generate_digest(page(N))

[0058] ifquery_hash_digest(page(N))then

[0059] transmit pageN_digest#transfor memory page hash digest to dest

[0060] else

[0061] transmit page[N]immediately#transfor memory page content to dest

[0062] update_client_page[N]#update hash digest and page digest to client

[0063] Similarly, if the accepted business logic is a hash digest, then the value of the corresponding digest is obtained through the client.

[0064] Example 2

[0065] This embodiment provides an electronic device, including: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0066] The processor executes the aforementioned method for deduplication of multi-virtual machine hot migration data in a cloud computing environment by calling computer programs stored in memory.

[0067] The electronic device can vary considerably depending on its configuration or performance. It may include one or more Central Processing Units (CPUs) and one or more memories, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the multi-virtual machine hot migration data deduplication method provided in the above-described embodiment. The electronic device may also include other components for implementing its functions; for example, it may have wired or wireless network interfaces and input / output interfaces for data input and output. Further details are omitted here.

[0068] Example 3

[0069] This embodiment proposes a computer-readable storage medium on which an erasable and rewritable computer program is stored.

[0070] When a computer program runs on a computer device, it causes the computer device to execute the aforementioned method for deduplication of multi-virtual machine hot migration data in a cloud computing environment.

[0071] For example, computer-readable storage media can be read-only memory (ROM), random access memory (RAM), compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage devices.

[0072] In summary, the purpose of this patent is to fully understand the virtual machine memory data model based on various image IDs in a cloud computing environment through system design and repeated hot migration tests. This involves forming a set of hash digest and memory data pairs, and caching the data in memory beforehand. When a virtual machine migrates, the system queries the cached data and, for matched migration data, only transmits the hash digest value. This effectively deduplicates the virtual machine migration data, reducing the overall virtual machine hot migration time. More importantly, this deduplication operation alleviates the traffic pressure on the business network switches caused by large-scale virtual machine migrations, and reduces the impact of virtual machine migration on other normally functioning cloud hosts within the environment.

[0073] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0077] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for deduplicating data during hot migration of multiple virtual machines in a cloud computing environment, characterized in that, The specific steps include: S1, by continuously learning from the hot migration of virtual machines in the test environment, mastering the hash digest->value pairs of virtual machine memory data based on the operating system image ID, and mastering the hit rate of the data; S2, by importing the hash digest->value pairs of virtual machine memory data based on the operating system image ID, which have a hit rate of over 95% learned in the test environment, into the migration cache server, the deduplication of multi-virtual machine hot migration data is completed. The deduplication of multi-virtual machine hot migration data includes: setting up a host A to be migrated and a host B to be migrated; creating a set of n virtual machines on the same open-source operating system image on the host A to be migrated, representing the nth virtual machine; at the start of the migration, the management platform sends a migration request to the host A to be migrated and determines the destination server to be migrated, the host B to be migrated; after receiving the migration request, the hypervisor of the host A to be migrated sends a query request to the client of the host A to query whether the hash digest and memory page data of the operating system are in the client's cache; if there is no corresponding data in the client's cache of the host A to be migrated, a request is sent to the migration cache server, and the migration cache server returns the hash digest value of the data with a hit rate of more than 95% in the operating system image to the host A to be migrated; the hypervisor of the host B to be migrated sends the same query request to the client of the host B to be migrated; if there is no corresponding data in the client of the host B to be migrated, the migration cache server returns the hash digest value of the data with a hit rate of more than 95% in the operating system image and the corresponding memory page data to the host B to be migrated.

2. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, The specific steps of S1 include: S11. In the test environment, based on the operating system image provided on the cloud platform, a large number of virtual machines are created using a single image, and the corresponding test business models are configured; S12. Repeated hot migration tests are performed on the virtual machines. While the virtual machines are migrating and copying memory, libvirtd calculates the hash digest value for each memory page in 4KB units using a hash algorithm, and temporarily stores the hash digest value and the corresponding memory data in the memory database on the host client side; S13. The host client side sends the memory page and its hash digest value to the migration cache server via the network. The server side of the migration cache server compares the hash digest value and checks whether the hash digest value already exists in the database. If it exists, the hit of the corresponding block is incremented by 1; if it does not exist, its hash digest and memory page data are stored; S14. Through repeated migration in the test environment, the data value on the server side of the migration cache server is continuously updated, and data with a hit rate of over 95% is extracted.

3. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, During the migration, memory data is copied. A hash digest value is calculated for the 4KB memory page data to be transferred. The client-side cache of the host to be migrated (A) is checked to see if the hash digest value exists. If the hash digest value exists, the host to be migrated (A) only transmits the hash digest value to the hypervisor process of the host to be migrated (B). If the hash digest value does not exist, the host to be migrated (A) transmits the complete data of the memory page to the host to be migrated (B) according to the original migration method. At the same time, the hash digest value and the corresponding memory page data are transferred to the client-side cache of the host to be migrated (B), and subsequently synchronized to the server side of the migration cache server.

4. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, If the hypervisor of the host B being migrated receives a hash digest value, then the corresponding hash digest value is searched in the client cache of the host B being migrated, the corresponding memory page data is obtained, and the memory data copying process is completed. The above memory data copying process is repeated until the entire memory copying phase is completed. Then, the virtual machine state is switched from the host A to the host B being migrated, and then the migration is performed. After the migration is completed, the above process is repeated until the entire set of virtual machines is migrated.

5. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, The client-side architecture includes: firstly, the client starts an in-memory database, Redis, to cache the hash digest values ​​of memory pages and the corresponding memory page data; the backend is used to handle internal business logic; and the frontend provides APIs for communication with the hypervisor and server.

6. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, The server-side structure includes: a MySQL database running on the server side, with one table created for each operating system on the cloud platform. Each table mainly stores five fields: hash, in-memory data, total number of migrations, number of migration hits, and migration hit rate; the backend is used to handle internal business logic; and the frontend provides APIs for communication with the hypervisor and client.

7. The method for deduplication of multi-virtual machine hot migration data in a cloud computing environment as described in claim 1, characterized in that, The hypervisor needs to modify the migration business logic. During the migration process, it first checks whether there is a hash digest for the memory page locally, sends pseudocode, and if the receiving business logic is a hash digest, it obtains the memory page data with the corresponding hash digest value through the client.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements a method for deduplication of multi-virtual machine hot migration data in a cloud computing environment, as claimed in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for deduplication of multi-virtual machine hot migration data in a cloud computing environment, as claimed in any one of claims 1-8.

Images