A container service hot migration platform and method in node failure avoidance scenario

Abstract

The present invention discloses a container service hot migration platform and method in a node failure avoidance scenario, belonging to the technical field of container service hot migration. The container service hot migration platform is based on a Kubernetes cluster, including a Master node and a Node node, the Master node includes a MigSS controller and a migration controller, the Node node includes a source working node and a target working node, the source working node and the target working node both include an extended Kubelet and a migration executor; the MigSS controller is responsible for assigning different management strategies to a source Pod and a cloned Pod; the migration controller is responsible for the overall control of stateful migration; the extended Kubelet has a container state archiving function; the migration executor is managed by the migration controller, is responsible for state archive file transmission, reception and establishment of a stateful mirror, and controls the incremental state synchronization of the source Pod and the cloned Pod in the synchronization stage; the platform and method provided by the present invention effectively reduce the migration time of an application, and achieve failure computing node avoidance through an active migration mechanism.

Description

Technical Field

[0001] The present invention belongs to the technical field of container service hot migration, and in particular, relates to a container service hot migration platform and method in a node failure avoidance scenario. Background Art

[0002] StatefulSet resource objects are mainly used to deploy stateful services in the Kubernetes environment. The StatefulSet resource object uses a replacement mechanism to manage failed Pods and provides automatic repair capabilities to a certain extent. However, it still has certain limitations in ensuring the high availability of stateful services:

[0003] ① When a stateful service uses the StatefulSet resource object to manage its service instance, the rescheduling of the Pod retains persistent data and network topology information. However, the information retained by this mechanism does not include the runtime information of the container application in the Pod. According to the CAP theory of distributed software, this will cause the Pod application that meets the CP characteristics to go through a long initialization phase after re-creation, or cause the loss of unwritten data of the Pod application that meets the AP characteristics, which brings great disadvantages. To avoid this problem, the Pod itself needs to be as stateless as possible, which will have certain requirements for the design of stateful services. Therefore, when the Pod is rescheduled, it is necessary to pay attention to the recovery of the runtime state of the container.

[0004] ② Since each Pod in the Pod set uses its own PV, has independent persistent data, and has a unique DNS resolution record, it has certain special features. This means that each Pod in the replica set is not necessarily a completely equal relationship. When a Pod fails, other Pods cannot seamlessly replace it, but must wait for the failed Pod to be scheduled to restart, which will result in a long period of degradation or service interruption.

[0005] ③When a node in a Kubernetes cluster is unexpectedly shut down due to extreme situations such as power outages or failures, stateful applications will not be rescheduled and created on other healthy nodes, but will remain on the shut down node. This is because the abnormal shutdown of the node causes the kubelet to fail to detect the shutdown event of the node, which ultimately results in the Pod on the node not being terminated and releasing resources. Kubernetes cannot determine when the node will resume and rejoin the cluster. In this uncertain event, the StatefulSet resource object will not create a new Pod to replace the Pod on the unresponsive node in order to prevent conflicts caused by the coexistence of multiple Pods with the same identity. Although in the latest solution, you can manually set a deactivation taint for the unresponsive node to force the deletion of resources, this requires the intervention of the administrator. If the administrator does not intervene in time, service availability will not be guaranteed.

[0006] The stateful migration mechanism copies and saves the information about the runtime state of the container application to form an archive file, then reads the archive file on another node and restores the information in the system. The restarted container application has the information and resources before the migration, so the continuity of the application is guaranteed. Currently, the Checkpoint / Restore technology is mainly used. Checkpoint / Restore technology has been widely used on virtual machines, and in recent years, the research on its application to containers has gradually increased. However, there are few studies on the application of this mechanism in Kubernetes. The main reason is that this conflicts with the container management concept of Kubernetes. Specifically: First, Kubernetes uses Pod as the smallest scheduling unit to manage containers, and hopes that the container is as stateless as possible, so that the container can be quickly restarted on another node when it fails. This means that the life cycle of the container is not expected to be affected by external components, and the use of Checkpoint / Restore technology for stateful migration destroys the life cycle of the container, which will add a "migrating" stage to the container. At the same time, the added lifecycle stages will not be compatible with replica set resource objects (such as Deployment, StatefulSet, etc.), which will cause the loss of Pod configuration status in the Kubernetes cluster and even cause conflicts; secondly, the stateful migration mechanism will involve the migration of more data. Although it ensures the continuity of program operation, it may not be better than the stateless migration mechanism in terms of time continuity. Summary of the invention

[0007] The purpose of the present invention is to provide a container service hot migration platform and method in a node failure avoidance scenario, so as to solve the problems in the prior art that the stateful service availability in the Kubernetes environment is poor and the life cycle of the Pod is easily destroyed during dynamic migration.

[0008] To achieve the above object, the present invention provides a container service hot migration platform in a node failure avoidance scenario. The container service hot migration platform is based on a Kubernetes cluster, including a Master node and a Node node. The Master node includes a MigSS controller and a migration controller. The Node node includes a source working node and a target working node. Both the source working node and the target working node include an extended Kubelet and a migration executor.

[0009] The MigSS controller is responsible for assigning different management policies to the source Pod and the cloned Pod;

[0010] The migration controller is responsible for the overall control of stateful migration;

[0011] The extended Kubelet has the container status archiving function;

[0012] The migration executor is managed by the migration controller and is responsible for transferring and receiving state archive files and creating stateful mirrors. It also controls the incremental state synchronization between the source Pod and the cloned Pod during the synchronization phase.

[0013] Preferably, when the migration controller receives a migration command from a node failure prediction component or a human, it obtains a list of Pods that need to be migrated, triggers the migration work, and notifies the Kubelet on the source working node to archive the status of the Pod to be migrated. After completion, the source Pod continues to provide services to the outside world.

[0014] Preferably, the migration executor is a daemon process that exists on each working node. The migration executor of the source working node transfers the state archive file to the target working node. The migration executor of the target working node uses the archive file to create a stateful image, and then the system starts the cloned Pod.

[0015] Preferably, there are two Pod groups in the control logic of the MigSS controller, namely, the Pods marked in normal use by the official group and the Pods cloned by the backup group; the cloned Pods are soft-managed by the controller, which captures the cloned Pods, adds OwnerReference to them and puts them under the management of the controller, but does not include them in the control loop, and does not configure network resources. Therefore, the cloned Pods are transparent in the microservice architecture. When a Pod marked in normal use by the official group is empty, the MigSS controller first checks whether the corresponding cloned Pod can be found in the backup group. If so, the cloned Pod will replace the source Pod without rescheduling and creating it, and network resources will be configured for the cloned Pod. The cloned Pod continues to provide services in the microservice architecture as the source Pod.

[0016] The present invention also provides a method for a container service hot migration platform in a node failure avoidance scenario, including a migration preparation stage and a batch migration stage; the migration preparation stage specifically includes the selection of a target working node and the pre-pulling of a basic image; the batch migration stage specifically includes network-aware batch migration.

[0017] Preferably, the specific process of selecting the target working node and pre-pulling the base image in the migration preparation phase is as follows:

[0018] Step 1: The migration controller obtains the Pod information to be migrated on the source working node, forms a pre-migration queue, and determines whether the pre-migration queue is empty. If it is empty, the process ends; otherwise, the process proceeds to step 2.

[0019] Step 2: Get the head Pod information, which contains the image layer set of the base image of the container in the Pod.

[0020] Step 3: Calculate the image layer similarity between the base image and the candidate nodes, where the candidate node set comes from the pre-selected node set of the default scheduler of the Kubernetes cluster;

[0021] Step 4: Repeat step 3 to determine whether all candidate node sets have been traversed. If not, continue to step 3. Otherwise, delete the target working nodes whose similarity is lower than the target similarity, and sort the remaining nodes by similarity to obtain the target working nodes and pull the image.

[0022] Step 5. Repeat steps 2, 3, and 4 until the pre-migration queue is empty.

[0023] Preferably, the calculation process of the image layer similarity between the base image and the candidate node in step 3 is as follows:

[0024] S31A, consider a base image as a set of image layers stacked and packaged together b, , , where each image layer has a unique hash ID, let the candidate node set be , Represents the set of all image layers on the i-th candidate node, , b and The degree of difference between them affects the cost of pulling the base image. Suppose the image layer set A certain image layer layer and image layer The inclusion relationship is:

[0025] ;

[0026] S32A, obtain the vector through the inclusion relationship in S31A and Representation:

[0027] ;

[0028] ;

[0029] S33A, calculate b and The similarity between them is calculated as follows:

[0030] .

[0031] Preferably, the process of network-aware batch migration in the batch migration phase is as follows:

[0032] S1. Get the migration queue. The Pod that has pre-pulled the base image is added to the migration queue.

[0033] S2, take out the head Pod information and determine whether the transfer set is empty. If the transfer list is empty, directly transfer the state archive and add it to the transfer set. Otherwise, enter S3.

[0034] S3. Before transferring the archived file of the current Pod, first calculate the network bandwidth resources and the transfer time, and determine whether the increase of the container batch K by 1 will cause the file in transfer to time out. If so, put it back into the migration queue and return to S2. Otherwise, determine whether the archive has timed out under the obtained network bandwidth resources. If so, put it back into the migration queue and return to S2. Otherwise, add it to the transfer set.

[0035] S4. If both indicators of S3 do not time out, add the state archive to the transmission set;

[0036] S5. Repeat S2, S3, and S4 until the migration queue is empty.

[0037] Preferably, the specific process of calculating bandwidth resources and transmission time and determining whether to join the transmission set is as follows:

[0038] S31B, let the size of each container snapshot file be During the migration process, the total bandwidth resources of the source host N is evenly distributed to containers migrated in parallel. The container batch is K. Then the network bandwidth resources used by each container during the migration process for:

[0039] ;

[0040] S32B, calculate transfer archive file Time required , the calculation expression is as follows:

[0041] ;

[0042] In the formula, For the data that has been transmitted, The time spent so far;

[0043] S33B, set constraints, specifically to ensure that the migration time of all snapshot files is within To ensure that the slowest batch migration time meets the total time limit, the calculation expression is as follows:

[0044] ;

[0045] S34B, the migration batch size satisfies the following formula:

[0046] ;

[0047] In the formula, The maximum size of the container snapshot file data at the current stage.

[0048] Therefore, the present invention adopts the above-mentioned container service hot migration platform and method in a node failure avoidance scenario, which has the following beneficial effects:

[0049] (1) Implemented stateful migration of container applications in the Kubernetes environment. In actual operation, "clone" was used instead of "migration" to protect the life cycle of the container.

[0050] (2) The resource object MigSS and its controller are designed to integrate the application's stateful migration capability with the stateful service replica set management capability. Compared with the StatefulSet resource object based on the replacement mechanism, this solution realizes comprehensive state migration and recovery;

[0051] (3) Based on the stateful migration of applications, the log playback mechanism is used to achieve dynamic migration of applications to shorten the downtime caused by migration;

[0052] (4) The present invention supports an active migration mechanism under the avoidance of server failures, and thus uses a similarity filtering mechanism to optimize the basic image pulling process, and supports batch parallel migration of Pods to shorten the total migration time and improve service availability assurance.

[0053] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is an overall architecture diagram of a container service hot migration platform in a node failure avoidance scenario of the present invention;

[0055] Figure 2 is a timing diagram of the migration process of an embodiment of the present invention;

[0056] Figure 3 A management principle diagram of a MigSS controller according to an embodiment of the present invention;

[0057] Figure 4 A diagram of a synchronization management architecture according to an embodiment of the present invention;

[0058] Figure 5 A schematic diagram of batch migration according to an embodiment of the present invention;

[0059] Figure 6 This is a flow chart of target node selection and base image pre-pulling according to an embodiment of the present invention;

[0060] Figure 7 The figure is a flow chart of batch migration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1-Figure 7 A container service hot migration platform for node failure avoidance scenarios. The container service hot migration platform is based on a Kubernetes cluster and includes a Master node (main node) and a Node node (worker node). The Master node includes a MigSS controller and a migration controller. The Node node includes a source work node and a target work node. Both the source work node and the target work node include an extended Kubelet and a migration executor.

[0063] The MigSS controller is responsible for assigning different management strategies to the source Pod and the cloned Pod; there are two Pod groups in the control logic of the MigSS controller, namely the Pod marked as being in normal use by the official group and the Pod marked as being cloned by the backup group; the cloned Pod is soft-managed by the controller, which captures the cloned Pod, adds OwnerReference to it and puts it under controller management, but does not include it in the control loop and does not configure network resources. Therefore, the cloned Pod is transparent in the microservice architecture. When a Pod marked as being in normal use by the official group is empty, the MigSS controller first checks whether the corresponding cloned Pod can be found in the backup group. If so, the cloned Pod will replace the source Pod without rescheduling and creating it, and network resources will be configured for the cloned Pod. The cloned Pod will continue to provide services in the microservice architecture as the source Pod.

[0064] The migration controller is responsible for the overall control of stateful migration. When the migration controller receives a migration command from the node failure prediction component or a manual command, it obtains the list of Pods that need to be migrated and triggers the migration work. The migration controller notifies the Kubelet on the source working node to archive the status of the Pod to be migrated. After completion, the source Pod continues to provide external services.

[0065] The extended Kubelet has the container status archiving function;

[0066] The migration executor is managed by the migration controller. It is responsible for transferring, receiving, and establishing stateful images of state archive files, and controls the incremental state synchronization of the source Pod and the cloned Pod during the synchronization phase. The migration executor is a daemon process that exists on each worker node. The migration executor of the source worker node transfers the state archive file to the target worker node. The migration executor of the target worker node uses the archive file to establish a stateful image, and then the system starts the cloned Pod.

[0067] The original stateful service resource object StatefulSet in the Kubernetes environment has limitations in service continuity and availability. This limitation is solved by integrating the stateful migration mechanism. The stateful migration mechanism allows the system to use the stateful image built by the source Pod to create a new clone Pod. The stateful image contains the runtime state of the container, such as process information and memory data, so that the clone Pod can continue to run from the recorded breakpoint, ensuring the continuity of the service at the application level. At the same time, in order to expand the capabilities of the stateful service resource object, we designed a resource object MigSS with migration management function, so that the clone Pod and the source Pod can be coordinated and managed. After the source Pod is destroyed, the clone Pod can enjoy cluster resources as the source Pod and provide external services, thereby ensuring the continuity of the service in the microservice architecture. In the entire design, we non-invasively added the required components based on the original Kubernetes environment. Among them, the migration controller is responsible for the overall control of the stateful migration, and the Checkpoint / Restore API interface is integrated on the Kubelet to become an extended Kubelet, so that the Kubelet has the function of archiving container status. The migration executor on each node is managed by the migration controller, which is responsible for transferring and receiving state archive files and establishing stateful mirrors, and controlling the incremental state synchronization between the source Pod and the cloned Pod during the synchronization phase. The MigSS controller is responsible for assigning different management policies to the source Pod and the cloned Pod. The specific migration steps are summarized as follows: Figure 2As shown. The migration controller is located on the Master node. When it receives a migration command from the node failure prediction component or a manual command, it obtains the list of Pods that need to be migrated and triggers the migration work. The migration controller notifies the Kubelet on the source node to archive the status of the Pod to be migrated. After completion, the source Pod continues to provide external services. The migration executor is a daemon process that exists on each working node. The migration executor of the source node transfers the state archive file to the target node. The migration executor of the target node uses the archive file to create a stateful mirror, and then the system starts the cloned Pod. When the cloned Pod is created, it enters the incremental state synchronization phase. At this time, the resource object controller MigSS needs to manage both Pods at the same time. Since the source Pod needs to continue to provide services to the outside world, and the cloned Pod only serves as a substitute to obtain the incremental state of the source Pod, the management strategies of the two are different. The management strategy of the MigSS controller is as follows Figure 3 As shown. During the migration cycle, MigSS captures the created cloned Pods and includes them in the standby Pod group. The migration controller synchronizes the incremental state generated by the source Pod during the migration for the cloned Pod. The access requests of the source Pod during the migration will be recorded, and the cloned Pod uses these records to supplement the incremental state. After the incremental state synchronization phase is over, the source Pod is deleted, and MigSS transfers the cloned Pod to the formal Pod group. After the checkpoint operation is performed on the source Pod and the archive file is recorded, the Pod officially enters the migration cycle, and the log of the source Pod needs to be recorded from this moment on. After the cloned Pod is created, multiple rounds of iterative synchronization operations are performed through the playback of these logs. The management architecture is as follows: Figure 4 As shown. MigSS adds a Sidecar proxy container to each Pod it manages, which is used as a request proxy for the main container. After entering the migration cycle, the proxy container records all requests for the main container, and then in the incremental state synchronization phase, the migration executor sends these records to the cloned Pod for playback. Many stateful applications use specific protocols for communication, such as the mysql protocol used by MySQL applications. Therefore, in addition to supporting common HTTP, gRPC and other protocols, the proxy container also supports special protocols used by stateful applications. This is solved by configuring or developing corresponding protocol filters for the proxy container.

[0068] A method for hot migration of container services in node failure avoidance scenarios. In node failure avoidance migration scenarios, there are always multiple Pods that need to be migrated on the source node (sub-healthy node). Parallel transmission of data from multiple Pods can save a lot of time compared to serial transmission, which is an important factor affecting the migration of service instances in server failure avoidance scenarios. In the migration scenario of large batches of service instances, there are multi-container and multi-stage migration processes, so we optimized these processes to reduce the total migration time. The batch migration process is as follows: Figure 5 Specifically, we advance the basic image pulling step required by the service instance to be migrated to before the formal migration, and adopt a parallel transmission mechanism when migrating multiple service instances. Specifically, it includes the migration preparation stage and the batch migration stage; the migration preparation stage is specifically the selection of the target working node and the pre-pulling of the basic image; the batch migration stage is specifically the network-aware batch migration.

[0069] During the stateful migration process, the base image occupies a considerable amount of data, but this data is not included in the state archive. The target node can choose to pull the image instead of accepting the transmission from the source node. Therefore, we separate the base image pull from the migration cycle and migration data. Before the service instance is migrated, we will first pull the base image on the target node. We call this process the migration preparation phase. The time taken by this phase should be as short as possible, otherwise it will squeeze the actual migration time.

[0070] Considering that scheduling in a Kubernetes cluster requires consideration of multiple factors, we do not simply select the node with the highest similarity as the target node, but select it in combination with the system's scheduling module. The process is summarized as follows: Figure 6 The specific process of selecting the target working node and pre-pulling the base image in the migration preparation phase is as follows:

[0071] Step 1: The migration controller obtains the Pod information to be migrated on the source working node, forms a pre-migration queue, and determines whether the pre-migration queue is empty. If it is empty, the process ends; otherwise, the process proceeds to step 2.

[0072] Step 2: Get the head Pod information, which contains the image layer set of the base image of the container in the Pod.

[0073] Step 3: Calculate the image layer similarity between the base image and the candidate node, where the candidate node set comes from the pre-selected node set of the default scheduler of the Kubernetes cluster; the calculation process of the image layer similarity between the base image and the candidate node is as follows:

[0074] S31A, We propose a similarity filtering strategy to filter the candidate set of target nodes for migration based on the similarity of the image layer. This method can minimize the amount of data pulled when the target node pulls the base image, resulting in minimal time and performance overhead. The similarity filtering strategy focuses on the image layer information and regards a base image as a set of image layers stacked and packaged. , , where each image layer has a unique hash ID Let the candidate node set be , Represents the set of all image layers on the i-th candidate node, , b and The degree of difference between them affects the cost of pulling the base image. Suppose the image layer set A certain image layer layer and image layer The inclusion relationship is:

[0075] ;

[0076] S32A, obtain the vector through the inclusion relationship in S31A and Representation:

[0077] ;

[0078] ;

[0079] S33A, calculate b and The similarity between them is calculated as follows:

[0080] .

[0081] Step 4: Repeat step 3 to determine whether all candidate node sets have been traversed. If not, continue to step 3. Otherwise, delete the target working nodes whose similarity is lower than the target similarity, and sort the remaining nodes by similarity to obtain the target working nodes and pull the image.

[0082] Step 5. Repeat steps 2, 3, and 4 until the pre-migration queue is empty.

[0083] When using stateful migration to avoid server failures, multiple Pods usually need to be migrated on the source host. To minimize losses, we need to migrate as many Pods as possible within a limited time, so we use parallel migration. Since checkpoint operations will cause the container to suspend service, in order to prevent batch checkpoint operations from causing large-scale service degradation, checkpoint operations are performed in serial mode, and then the checkpoint archive files are transferred in parallel. If the batch size of the parallel migration is too small, the total multi-container migration time will be extended; if the batch size of the parallel migration is too large, the transmission speed will be too slow, resulting in a single container migration phase that takes too long, and too many source Pod logs accumulate, affecting the convergence time of the synchronization phase. We use the longest transmission time for a single archive. Make restrictions.

[0084] The process of network-aware batch migration in the batch migration phase is as follows: Figure 7 As shown, the details are as follows:

[0085] S1. Get the migration queue. The Pod that has pre-pulled the base image is added to the migration queue.

[0086] S2, take out the head Pod information and determine whether the transfer set is empty. If the transfer list is empty, directly transfer the state archive and add it to the transfer set. Otherwise, enter S3.

[0087] S3. Before transferring the archived file of the current Pod, first calculate the network bandwidth resources and the transfer time, and determine whether the increase of the container batch K by 1 will cause the file in transfer to time out. If so, put it back into the migration queue and return to S2. Otherwise, determine whether the archive has timed out under the obtained network bandwidth resources. If so, put it back into the migration queue and return to S2. Otherwise, add it to the transfer set.

[0088] S4. If both indicators of S3 do not time out, add the state archive to the transmission set;

[0089] S5. Repeat S2, S3, and S4 until the migration queue is empty.

[0090] The specific process of calculating bandwidth resources and transmission time and determining whether to join the transmission set is as follows:

[0091] S31B, let the size of each container snapshot file be , that is, the amount of data per container, during the migration process, the total bandwidth resources of the source host N is evenly distributed to containers migrated in parallel. The container batch is K. Then the network bandwidth resources used by each container during the migration process for:

[0092] ;

[0093] S32B, calculate transfer archive file Time required , the calculation expression is as follows:

[0094] ;

[0095] In the formula, For the data that has been transmitted, The time spent so far;

[0096] S33B, set constraints, specifically to ensure that the migration time of all snapshot files is within To ensure that the slowest batch migration time meets the total time limit, the calculation expression is as follows:

[0097] ;

[0098] S34B, the migration batch size satisfies the following formula:

[0099] ;

[0100] In the formula, The maximum size of the container snapshot file data at the current stage.

[0101] This parallel batch migration method based on network resource awareness shortens the total migration time of multiple Pods on the source node. The shorter total migration time greatly reduces the risk of Pod application failure caused by sub-healthy nodes, thereby ensuring the service availability of microservices.

[0102] Therefore, the present invention adopts the above-mentioned container service hot migration platform and method in the node failure avoidance scenario, based on the custom resource object and object controller mechanism, the set migration controller and the migration daemon realize the hot migration of containers in the Kubernetes environment, and realizes the comprehensive state migration and recovery of cloud native applications at the container level and the microservice architecture level, and realizes the avoidance of faulty computing nodes through an active migration mechanism. At the same time, for the scenario of avoiding faulty computing nodes, it proposes to select the target working node and pre-pull the image based on the image layer similarity filtering mechanism, and supports batch parallel migration of network resources, which effectively reduces the migration time of the application.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for hot migration of container services in a node failure avoidance scenario, characterized in that: It includes the migration preparation phase and the batch migration phase; The migration preparation phase specifically involves the selection of target working nodes and pre-pulling of base images; the batch migration phase specifically involves network-aware batch migration; The specific process of selecting the target working node and pre-pulling the base image in the migration preparation phase is as follows: Step 1: The migration controller obtains the Pod information to be migrated on the source working node, forms a pre-migration queue, and determines whether the pre-migration queue is empty. If it is empty, the process ends; otherwise, the process proceeds to step 2. Step 2: Get the head Pod information, which contains the image layer set of the base image of the container in the Pod. Step 3: Calculate the image layer similarity between the base image and the candidate nodes, where the candidate node set comes from the pre-selected node set of the default scheduler of the Kubernetes cluster; Step 4: Repeat step 3 to determine whether all candidate node sets have been traversed. If not, continue to step 3. Otherwise, delete the target working nodes whose similarity is lower than the target similarity, and sort the remaining nodes by similarity to obtain the target working nodes and pull the image. Step 5: Repeat steps 2, 3, and 4 until the pre-migration queue is empty. The calculation process of the image layer similarity between the base image and the candidate node in step 3 is as follows: S31A, consider a base image as a set of image layers stacked and packaged together b, , , where each image layer has a unique hash ID Let the candidate node set be , Represents the set of all image layers on the i-th candidate node, , b and The degree of difference between them affects the cost of pulling the base image. Suppose the image layer set A certain image layer layer and image layer The inclusion relationship is: ； S32A, obtain the vector through the inclusion relationship in S31A and Representation: ； ； S33A, calculate b and The similarity between them is calculated as follows: ； The process of network-aware batch migration in the batch migration phase is as follows: S1. Get the migration queue. The Pod that has pre-pulled the base image is added to the migration queue. S2, take out the head Pod information and determine whether the transfer set is empty. If the transfer list is empty, directly transfer the state archive and add it to the transfer set. Otherwise, enter S3. S3. Before transferring the archived file of the current Pod, first calculate the network bandwidth resources and the transfer time, and determine whether the increase of the container batch K by 1 will cause the file in transfer to time out. If so, put it back into the migration queue and return to S2. Otherwise, determine whether the archive has timed out under the obtained network bandwidth resources. If so, put it back into the migration queue and return to S2. Otherwise, add it to the transfer set. S4. If both indicators of S3 do not time out, add the state archive to the transmission set; S5. Repeat S2, S3, and S4 until the migration queue is empty. The specific process of calculating bandwidth resources and transmission time and determining whether to join the transmission set is as follows: S31B, let the size of each container snapshot file be During the migration process, the total bandwidth resources of the source host N is evenly distributed to containers migrated in parallel. The container batch is K. Then the network bandwidth resources used by each container during the migration process for: ； S32B, calculate transfer archive file Time required , the calculation expression is as follows: ； In the formula, For the data that has been transmitted, The time spent so far; S33B, set constraints, specifically to ensure that the migration time of all snapshot files is within To ensure that the slowest batch migration time meets the total time limit, the calculation expression is as follows: ； S34B, the migration batch size satisfies the following formula: ； In the formula, The maximum size of the container snapshot file data at the current stage; The container service hot migration platform is based on the Kubernetes cluster, including the Master node and Node nodes. The Master node includes the MigSS controller and the migration controller. The Node nodes include the source working node and the target working node. Both the source working node and the target working node include the extended Kubelet and the migration executor. The MigSS controller inherits the capabilities of the Kubernetes replica set object and adds the ability to assign different management policies to the source Pod and the cloned Pod. The migration controller is responsible for the overall control of stateful migration; The extended Kubelet has the container status archiving function; The migration executor is managed by the migration controller and is responsible for transferring and receiving state archive files and creating stateful mirrors. It also controls the incremental state synchronization between the source Pod and the cloned Pod during the synchronization phase. When the migration controller receives a migration command from the node failure prediction component or a human, it obtains the list of Pods that need to be migrated and triggers the migration work. The migration controller notifies the Kubelet on the source working node to archive the status of the Pod to be migrated. After completion, the source Pod continues to provide external services. The migration executor is a daemon process that exists on each worker node. The migration executor of the source worker node transfers the state archive file to the target worker node. The migration executor of the target worker node uses the archive file to create a stateful image, and then the system starts the cloned Pod. There are two Pod groups in the control logic of the MigSS controller, namely the Pods marked as being in normal use by the official group and the Pods marked as being cloned by the backup group. The cloned Pods are soft-managed by the controller, which captures the cloned Pods, adds OwnerReference to them and puts them under the management of the controller, but does not include them in the control loop and does not configure network resources. When a Pod marked as being in normal use by the official group is empty, the MigSS controller first checks whether the corresponding cloned Pod can be found in the backup group. If so, the cloned Pod will replace the source Pod without rescheduling and creating it. At the same time, network resources are configured for the cloned Pod, and the cloned Pod continues to provide services in the microservice architecture as the source Pod.

Images