Secure container migration method, device, storage medium, and program product

Abstract

The embodiment of the invention provides a secure container migration method and device, a storage medium and a program product. In the embodiment of the invention, in the process of carrying out thermal migration on the first safety container, the target resource object and the non-target resource object are distinguished, different migration modes are adopted for the non-target resource object and the target resource object, and the migration processes of the non-target resource object and the target resource object are fused, namely, the cold migration process of the target resource object is carried out. According to the method, the target resource object is subjected to cold migration and fused in the hot migration process of the non-target resource object, so that the non-target resource object can normally operate in the cold migration process of the target resource object, the unavailable time generated by migration of different resource objects can be optimized, the unavailable time of each resource object in the migration process is reduced as much as possible, and the user experience is improved. In this way, rapid migration of the workload in the security container can be completed, and the problem of migration of the security container with the target resource object by a cloud manufacturer is solved.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202411764073.8 (22) Application Date 2024.12.03 (71) Applicant Cloud Intelligent Asset Holdings (Singapore) Pte Ltd Address 51 Bras Basa Road, Singapore, 1 Laizam Tower #03-06 (72) Inventors Bian Zizheng Liu Jiang (74) Patent Agency Beijing Taihe Jiusi Intellectual Property Agency Co., Ltd. 11610 Patent Attorney Zhang Ai Liu Ge (51) Int.Cl. G06F 9 / 455 (2018.01) (54) Invention Title Secure Container Migration Method, Device, Storage Medium and Program Product (57) Abstract This application provides a secure container migration method, device, storage medium and program product. In this embodiment, during the hot migration of the first secure container, a distinction is made between target and non-target resource objects. Different migration methods are used for target and non-target resource objects, and their migration processes are merged. Specifically, the cold migration process of the target resource object is integrated into the hot migration process of the non-target resource object. This ensures that non-target resource objects can operate normally during the cold migration of the target resource object, optimizing the downtime caused by migration for different resource objects and minimizing the downtime of each resource object during migration to improve user experience. This approach enables rapid migration of workloads within the secure container, solving the migration problem for cloud vendors of secure containers containing target resource objects. Claims 4 pages, Description 35 pages, Drawings 9 pages, CN 122152423 A 2026.06.05 CN 1 22 15 24 23 A 1. A secure container migration method, characterized in that it is used to migrate workloads in a first secure container on a source node, the first secure container including a target resource object and non-target resource objects, the target resource object being a pass-through device of the first secure container, the method comprising: during the operation of the first secure container, hot-migrating first state data generated by the non-target resource object of the first secure container to a non-target resource object in a second secure container, the second secure container being a secure container pre-created on the target node; pausing the operation of the target resource object of the first secure container, and cold-migrating the state data generated by the target resource object of the first secure container to the target resource object of the second secure container; pausing the operation of the non-target resource object of the first secure container, and migrating second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container.2. The method according to claim 1, wherein the target resource object and non-target resource object of the first secure container are mounted on a first virtual machine of the first secure container, and the first virtual machine includes a first hot-migrating component; during the operation of the first secure container, hot-migrating the first state data generated by the non-target resource object of the first secure container to the non-target resource object in the second secure container includes: during the operation of the first secure container, sending the first state data generated by the non-target resource object of the first secure container to a second hot-migrating component through the first hot-migrating component, so that the second hot-migrating component writes the first state data into the non-target resource object in the second secure container; wherein the second hot-migrating component is a hot-migrating component in the second virtual machine of the second secure container. 3. The method according to claim 2, wherein the first hot migration component comprises: a hot migration agent component located in the user space of the first virtual machine, and a hot migration driver component located in the guest operating system of the first virtual machine; during the operation of the first secure container, the first hot migration component gradually sends the first state data generated by the non-target resource object of the first secure container to the second hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container, comprising: during the operation of the first secure container, calling the hot migration driver component to obtain the first state data generated by the non-target resource object of the first secure container, and reporting it to the hot migration agent component; controlling the hot migration agent component to send the first state data to the second hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container. 4. The method according to claim 3, wherein calling the hot migration driver component to gradually obtain the first state data generated by the non-target resource object of the first secure container comprises: calling the hot migration driver component to obtain the memory data generated by the non-target resource object of the first secure container as the first state data. 5. The method according to any one of claims 1-4, characterized in that the target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the first secure container further includes: a first virtualization management module, the first virtualization management module including: a first migration management component; migrating the second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container includes: obtaining the second state data through the first migration management component, and sending the second state data to the first secure container.A second migration management component is provided for the second migration management component to write the second state data into the non-target resource object of the second secure container; wherein the second migration management component is a migration management component in the second virtualization management module of the second secure container. 6. The method according to any one of claims 1-4, wherein pausing the operation of the non-target resource object of the first secure container includes: pausing the operation of the first virtual machine through the first virtualization management module to pause the operation of the non-target resource object of the first secure container. 7. The method according to any one of claims 1-4, wherein the target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the guest operating system of the first virtual machine includes: a driver for the target resource object; pausing the operation of the target resource object of the first secure container and cold migrating the state data generated by the target resource object of the first secure container to the target resource object of the second secure container includes: pausing the operation of the target resource object of the first secure container through the driver of the target resource object and obtaining the state data of the target resource object of the first secure container; migrating the state data of the target resource object of the first secure container to the target resource object of the second secure container. 8. The method according to any one of claims 1-4, characterized in that it further comprises: pausing the operation of the first secure container through the container runtime component corresponding to the first secure container, the container runtime component being used to create and manage the first secure container; and / or obtaining the metadata of the first secure container through the container runtime component corresponding to the first secure container, and synchronizing the status data of the first secure container to the second secure container. 9. A secure container migration method, characterized in that it is used to migrate the workload of a first secure container on a source node to a pre-created second secure container on a target node, the second secure container including a target resource object and a non-target resource object, the target resource object being a pass-through device of the second secure container; the method includes: receiving first status data generated by a non-target resource object of the first secure container and storing the first status data in a non-target resource object of the second secure container; receiving status data generated by a target resource object of the first secure container and storing the status data of the target resource object of the first secure container in a target resource object of the second secure container, wherein the status data of the target resource object of the first secure container is transmitted while the target resource object of the first secure container is suspended; and receiving second status data generated by a non-target resource object of the first secure container and storing the second status data in a non-target resource object of the second secure container;Wherein, the second state data includes at least the state data generated by the non-target resource object of the first secure container during the transmission of the state data of the target resource object of the first secure container, and the second state data is transmitted while the target resource object of the first secure container is suspended. 10. The method according to claim 9, wherein the second secure container includes: a second virtual machine and a third virtual machine, the target resource object and the non-target resource object of the second secure container are respectively mounted on the second virtual machine and the third virtual machine, and the second virtual machine has access rights to the non-target resource object mounted on the third virtual machine; receiving the first state data generated by the non-target resource object of the first secure container and storing the first state data in the non-target resource object of the second secure container includes: the second virtual machine receives the first state data and, based on the access rights, saves the first state data to the non-target resource object mounted on the third virtual machine. 11. The method according to claim 10, wherein the second virtual machine includes a second hot migration component, and the second virtual machine receives the first state data and saves the first state data to a non-target resource object mounted on the third virtual machine based on the access permission, comprising: receiving the first state data sent by a first hot migration component through the second hot migration component, and saving the first state data to a non-target resource object mounted on the third virtual machine based on the access permission; wherein the first hot migration component is a hot migration component in the first virtual machine of the first secure container. 12. The method according to claim 11, wherein the second hot migration component includes: a hot migration agent component located in the user space of the second virtual machine, and a hot migration driver component located in the guest operating system of the second virtual machine; receiving the first state data sent by the first hot migration component through the second hot migration component, and saving the first state data to a non-target resource object mounted on the third virtual machine based on the access permission, comprising: receiving the first state data sent by the first hot migration component through the hot migration agent component; and, based on the access permission, calling the hot migration driver component to save the first state data to a non-target resource object mounted on the third virtual machine. 13. The method according to claim 10, wherein storing the state data of the target resource object of the first security container in the target resource object of the second security container comprises: storing the state data of the target resource object of the first security container in the target resource object mounted on the second virtual machine;The target resource object mounted on the second virtual machine is remounted to the third virtual machine to migrate the state data of the target resource object of the first secure container to the target resource object of the second secure container. 14. The method according to any one of claims 9-13, wherein the second secure container further comprises: a second virtualization management module, the second virtualization management module comprising: a second migration management component; receiving second state data generated by a non-target resource object of the first secure container and storing the second state data in a non-target resource object of the second secure container, comprising: using the second migration management component to receive second state data sent by a first migration management component, and storing the second state data in a non-target resource object mounted on the third virtual machine; wherein the first migration management component is a migration management component in the first virtualization management module of the first secure container. 15. The method according to any one of claims 9-13, characterized in that it further comprises: receiving metadata of the first secure container and synchronizing it to the second secure container through a container runtime component corresponding to the second secure container; and / or, running the second secure container through a container runtime component corresponding to the second secure container, and restoring the working state of the non-target resource object and the target resource object of the second secure container according to the state data in the non-target resource object and the target resource object of the second secure container, respectively. 16. A computer device, characterized in that it can be implemented as a source node, wherein a first secure container is deployed on the source node, the first secure container including a target resource object and a non-target resource object; the target resource object is a pass-through device of the first secure container, wherein the first secure container is used to perform the steps in the method according to any one of claims 1-8; and / or, it can be implemented as a target node, wherein a second secure container is deployed on the target node, the second secure container including a target resource object and a non-target resource object; the target resource object is a pass-through device of the second secure container, wherein the second secure container is used to perform the steps in the method according to any one of claims 9-15. 17. A computer-readable storage medium, characterized in that, when the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-8 and 9-15. 18. A computer program product, characterized in that it comprises a computer program / instructions, which, when executed by a processor, perform the steps of the method according to any one of claims 1-8 and 9-15. 19. A cloud computing system, characterized in that it comprises: a source node for performing the method according to any one of claims 1-8.The source node and the target node are used to perform the method according to any one of claims 9-15; the source node and the target node cooperate with each other to migrate the workload in the first secure container on the source node to the second secure container on the target node. Claims 4 / 4 Page 5 CN 122152423 A Secure Container Migration Method, Device, Storage Medium and Program Product Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a secure container migration method, device, storage medium and program product. Background Art

[0002] In cloud computing environments, multi-tenant architecture is usually adopted on a single server node. In order to achieve secure isolation between different tenants, containerization technology can be used to package applications and their dependencies into independent containers to ensure the security and consistency of applications. As a branch of container technology, secure containers aim to combine the security of virtualization and the convenience of containers to make the isolation between containers more thorough, and can provide a higher level of isolation and protection for applications in containers, and are therefore widely used.

[0003] As an important application scenario of cloud computing, AI (Artificial Intelligence) intelligent computing is increasingly deployed in secure containers, leading to a growing demand for computing power resources in these containers. GPUs (Graphics Processing Units), as a high-performance computing resource provided by cloud vendors, have demonstrated powerful performance in the fields of AI and deep learning. Therefore, some cloud vendors have proposed using GPUs in secure containers as the computing power supply for applications within them.

[0004] Combining GPUs with secure containers can solve the computing power demand problem of secure containers while achieving secure isolation. However, this also presents new technical challenges. For example, in cases of application upgrades, resource optimization, or application failures, it is necessary to migrate secure containers. How to migrate secure containers with GPU resources is a pressing technical problem. Summary of the Invention

[0005] This application provides a secure container migration method, device, storage medium, and program product to solve the migration problem of secure containers with pass-through devices (e.g., GPUs).

[0006] This application provides a secure container migration method for migrating workloads in a first secure container on a source node. The first secure container includes target resource objects and non-target resource objects. The target resource object is a pass-through device of the first secure container. The method includes: during the operation of the first secure container, hot-migrating first state data generated by the non-target resource object of the first secure container to a non-target resource object in a second secure container.In the target resource object, the second secure container is a secure container pre-created on the target node; the operation of the target resource object of the first secure container is paused, and the state data generated by the target resource object of the first secure container is cold-migrated to the target resource object of the second secure container; the operation of the non-target resource object of the first secure container is paused, and the second state data generated by the non-target resource object of the first secure container is migrated to the non-target resource object of the second secure container.

[0007] This application embodiment also provides a secure container migration method for migrating the workload of a first secure container on a source node to a second secure container pre-created on a target node, the second secure container including a target resource object and a non-target resource object; the target resource object is a pass-through device of the second secure container; the method includes: receiving first state data generated by the non-target resource object of the first secure container, and storing the first state data in the specification 1 / 35 page 6 CN 122152423 A The following are described: A non-target resource object exists in the second security container; status data generated by the target resource object of the first security container is received, and the status data of the target resource object of the first security container is stored in the target resource object of the second security container, wherein the status data of the target resource object of the first security container is transmitted while the target resource object of the first security container is suspended; and a second status data generated by the non-target resource object of the first security container is received, and the second status data generated by the non-target resource object of the first security container is stored in the non-target resource object of the second security container, wherein the second status data includes at least the status data generated by the non-target resource object of the first security container during the transmission of the status data of the target resource object of the first security container, and the second status data is transmitted while the target resource object of the first security container is suspended.

[0008] This application embodiment also provides a computer device, which can be implemented as a source node, on which a first secure container is deployed. The first secure container includes a target resource object and a non-target resource object. The target resource object is a pass-through device of the first secure container, wherein the first secure container is used to execute the steps in the secure container migration method; and / or, it can be implemented as a target node, on which a second secure container is deployed. The second secure container includes a target resource object and a non-target resource object. The target resource object is a pass-through device of the second secure container, wherein the second secure container is used to implement the steps in the secure container migration method.

[0009] This application embodiment also provides a computer-readable storage medium, which, when the computer program is executed by a processor...When the processor is executed, it enables the processor to implement the steps in the secure container migration method.

[0010] This application embodiment also provides a computer program product, including a computer program / instruction, which, when executed by a processor, performs the steps in the secure container migration method.

[0011] This application embodiment also provides a cloud computing system, including: a source node for executing the first set of secure container migration methods described above, and a target node for executing the second set of secure container migration methods described above; the source node and the target node cooperate with each other to migrate the workload in the first secure container on the source node to the second secure container on the target node.

[0012] In this embodiment of the application, during the hot migration of the first secure container, a distinction is made between target resource objects and non-target resource objects, and different migration methods are adopted for non-target resource objects and target resource objects. Furthermore, the migration processes of the two are merged, that is, the cold migration process of the target resource object is merged into the hot migration process of the non-target resource object. Thus, during the cold migration of the target resource object, the non-target resource object can operate normally, which can optimize the downtime caused by migration of different resource objects, minimizing the downtime of each resource object during the migration process and improving user experience. In this way, the rapid migration of workloads in the secure container can be completed, solving the migration problem of secure containers with target resource objects for cloud vendors. In addition, this method can better complete the rapid migration of workloads in the secure container, solving the migration problem of secure containers with target resource objects for cloud vendors. Brief Description of the Drawings

[0013] The accompanying drawings, which are included to provide a further understanding of this application and constitute a part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an improper limitation of this application. In the accompanying drawings:

[0014] FIG1a is a system architecture diagram of a computer device carrying a secure container provided in an exemplary embodiment of the present application;

[0015] FIG1b is another system architecture diagram of a computer device carrying a secure container provided in an exemplary embodiment of the present application (page 2 / 35, CN 122152423 A);

[0016] FIG2a is a schematic diagram of the structure of a source node provided in an exemplary embodiment of the present application;

[0017] FIG2b is a schematic diagram of the interaction process of state transition between a source node and a target node provided in an exemplary embodiment of the present application;

[0018] FIG3a is a schematic diagram of the structure of another source node provided in an exemplary embodiment of the present application;

[0019] FIG3b is a schematic diagram of the structure of a target node provided in an exemplary embodiment of the present application;

[0020] FIG3c is a schematic diagram of the interaction process of state transition between a source node and a target node provided in an exemplary embodiment of the present application;

[0021] Figure 4 is a flowchart illustrating another secure container migration method provided in an exemplary embodiment of this application;

[0022] Figure 5 is a flowchart illustrating another secure container migration method provided in an exemplary embodiment of this application;

[0023] Figure 6 is a schematic diagram illustrating a secure container migration device provided in an exemplary embodiment of this application;

[0024] Figure 7 is a schematic diagram illustrating a secure container migration device provided in an exemplary embodiment of this application;

[0025] Figure 8 is a schematic diagram illustrating a computer device provided in an exemplary embodiment of this application. Detailed Description

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entry points for users to choose to authorize or refuse. In addition, the various models involved in this application (including but not limited to language models or large models) comply with the relevant laws and standards.

[0028] With the deployment and application of AI applications and AI frameworks in secure containers, the requirements of secure containers for computing resources are getting higher and higher. Among them, AI frameworks refer to frameworks used to develop and deploy AI applications, such as TensorFlow (an open source machine learning platform), PyTorch (an open source machine learning library) or Caffe (a deep learning framework that combines expressiveness, speed and modular thinking); AI applications refer to applications developed and deployed based on AI frameworks, such as AI-based image processing programs, video processing programs or game programs, etc.

[0029] A secure container is a secure and isolated container runtime environment provided by virtualization technology. Unlike traditional container runtime environments, a secure container is a container runtime environment with a complete operating system (i.e., a guest operating system). Traditional container runtime environments do not have a guest operating system and rely on the host machine's operating system. Based on this difference, secure containers can offer higher security and isolation compared to traditional containers.

[0030] For example, Figure 1a is a system architecture diagram of a computer device 1 carrying a secure container. This computer device 1...This device can be implemented as any electronic device, such as a mobile phone, tablet computer, calculator, or any type of server; this embodiment is not limited to any particular type. As shown in Figure 1a, the computer device 1 may include a host machine, which includes the host machine's user space 11 (also referred to as the host machine's user mode), the host machine's operating system 12 (also referred to as the host machine's kernel mode), and a hardware layer 13. The hardware layer 13 includes various hardware resources of the host machine, such as, but not limited to, a GPU 131, a CPU (Central Processing Unit) 132, and a network interface card (NIC). In Figure 1a, the NIC is implemented as an RDMA (Remote Direct Memory Access) NIC 133 as an example, but it is not limited to this; the NIC can also be implemented using a regular NIC. The host machine's operating system 12 refers to the operating system running on the host machine, which is the foundation of secure containers and virtualization technology. It may include drivers for various hardware resources on the host machine, as well as a virtualization manager. The virtualization manager can be, but is not limited to, KVM (Kernel-based Virtual Machine). Figure 1a illustrates this using KVM 121 as an example. KVM allows the host operating system 12 to become a virtual machine monitor, enabling users to run multiple virtualization instances on the host operating system. The host's user space 11 is used to host various virtualization instances running on the host, such as secure containers 111 and container runtime components 112.

[0031] Secure containers 111 can implement their core functions and features through container runtime components 112. Container runtime components 112 are a series of software components responsible for managing secure containers in a containerized environment. These components work together to provide functions such as creating, starting, stopping, and monitoring secure containers 111, and ensure that secure containers can run securely in an isolated environment. In this embodiment, the implementation method of container runtime components 112 is not limited; for example, but not limited to, Docker, RunC, RunD, etc. Docker is one of the more popular container runtimes, providing complete container management functions. RunC is a lightweight container runtime that forms the basis of Docker and other container tools. RunD is a lightweight, secure container solution that provides a secure environment for running containerized applications, designed to offer enhanced security and isolation to protect sensitive data and applications from potential threats. RunD comprises several key components, one of which is the RunD VMM (Virtual Machine).Manager (Virtual Machine Manager), is a lightweight virtual machine management program used to manage and run virtual machine instances.

[0032] The secure container 111 may include: an application 1111 and a guest operating system 1112; the guest operating system 1112 in the secure container 111 represents an independent and isolated running environment, which may be a virtual machine, but is not limited thereto. Among them, the application 1111 may be implemented as the AI ​​framework or AI application mentioned above, or as any other type of application framework or application, which is not limited in this embodiment. Relative to the guest operating system 1112, the application 1111 may be located in the user space of the secure container 111. It should be noted that the user space 11 of the secure container 111 may include other components in addition to the application 1111.

[0033] The guest operating system 1112 refers to the operating system running inside the secure container 111. It can provide fine-grained access control functions, encrypted data storage functions, and trusted execution environment, etc., to ensure that the applications and data inside the container are protected. The access control function can restrict the process access to resources within the container based on security policies. The encrypted data storage function can protect the data inside the container based on encryption technology. The trusted execution environment can be used to provide a higher level of security protection. The guest operating system 1112 includes drivers for various hardware resources contained in the secure container, such as drivers that pass through to various hardware resources in the secure container using pass-through technology.

[0034] In this embodiment of the application, introducing GPU resources into the secure container scenario can solve the problem of the secure container's demand for computing power resources. It should be noted that, in addition to GPU resources, similar resources such as NPU (Neural Processing Unit) and DPU (Data Processing Unit) can also be used as computing power resources for the secure container. In this application embodiment, resources such as GPU, NPU, and DPU are collectively referred to as target resource objects. First, target resource objects refer to hardware computing resources. Second, target resource objects refer to computing resources that do not support virtualization themselves or for which there is currently no mature and standardized virtualization solution. For example, TPU (Tensor Processing Unit, a simple matrix multiplication coprocessor directly connected to the host motherboard) does not have built-in scheduling functions and therefore does not support virtualization. As another example, GPUs are mainly responsible for handling complex computing tasks and have relatively large built-in graphics memory. Although some virtualization solutions have emerged, most of them face challenges in resource allocation in multi-tenant environments and security issues between different tenant information, and are still immature.Furthermore, it is not standardized. In contrast, resource objects such as CPU, network card, and memory have relatively mature and standardized virtualization solutions, and these resource objects are referred to as non-target resource objects.

[0035] In the embodiments of this application, the secure container (or the cloud vendor to which the secure container belongs) has certain control requirements for resource objects. These control requirements refer to the control related to the secure container. For example, creating a secure container, destroying a secure container, and migrating a secure container all involve the control of related resource objects. Among them, the implementation method of controlling resource objects from the secure container level will be different depending on whether the resource object supports virtualization technology or whether there is a mature virtualization technology. For resource objects that support virtualization technology and have mature virtualization technology, such as CPU, network card, and memory, the cloud vendor to which the secure container belongs can implement the control logic for these resource objects in the container runtime component 112 in the secure container, without relying on the drivers of these resource objects. That is, the control logic for resource objects from the secure container level does not need to be implemented in the driver of the resource object. In this way, the cloud vendor to which the secure container belongs can implement the control logic for these resource objects on its own, without relying on the vendor of the resource object. For resource objects that do not support virtualization technology or lack mature virtualization technology, such as GPUs and TPUs, managing these resource objects from the secure container level requires relying on the resource object's driver, i.e., calling the relevant API interfaces exposed by the driver. Thus, the cloud vendor to which the secure container belongs cannot implement the management logic for these resource objects themselves and must rely on the device vendor's implementation of the driver.

[0036] Based on the above, the various resource objects involved in the secure container can be divided into two categories: target resource objects and non-target resource objects. Non-target resource objects refer to resource objects that can implement management logic in the container runtime component 112 within the secure container based on virtualization technology. Managing non-target resource objects from the secure container level does not depend on the driver for non-target resource objects. Target resource objects refer to resource objects that cannot implement management logic in the container runtime component 112 within the secure container based on virtualization technology. Managing target resource objects from the secure container level depends on the driver for the target resource object.

[0037] Because the target resource objects themselves do not support virtualization or the virtualization scheme is immature, target resource objects cannot be directly applied to the secure container 111 based on virtualization methods like resources such as CPUs and memory. This raises the primary question of how to apply the target resource object to the secure container 111. Therefore, this application provides two implementation methods:

[0038] Implementation Method 1: Extend the functionality of the driver for the target resource object (e.g., GPU) to add features for applying the target resource object to the secure container 111.The virtualization module (hereinafter referred to as the virtualization module) is used to virtualize the target resource object (such as GPU). The virtualization module divides the target resource object (such as GPU) into multiple virtual computing resources (virtual GPUs) and provides the divided virtual computing resources to the secure container 111. The driver of the target resource object is located in the host operating system 12 and needs to provide an API (Application Programming Interface) to the outside world. In this way, the secure container 111 can obtain the relevant status information of the target resource object by calling the API interface, which provides conditions for the migration of the secure container 111. However, since the driver of the target resource object is located in the host operating system, this involves interaction with the host kernel. This interaction process can increase the consumption of computing power and there is a problem of wasted computing power. In addition, since the driver is a proprietary solution of the target resource object vendor, it is difficult for the cloud vendor to control the technical details and to extend the additional functions of the driver.

[0039] Implementation method 2: The target resource object is directly connected to the secure container 111 using pass-through technology, forming an overall architecture of secure container 111 + pass-through device. In the use of passthrough technology, the target resource object can be referred to as the passthrough device of the secure container 111. Passthrough technology allows the secure container 111 to directly access the target resource object, instead of indirectly accessing it through the host operating system 12 (page 5 / 35 of the specification, CN 122152423 A). Since passthrough technology allows the secure container 111 to directly access the target resource object, it reduces the overhead of the host operating system 12 and improves I / O (Input / Output) performance; simultaneously, it reduces the number of times data is copied between the host operating system 12 and the secure container 111, reducing latency and increasing throughput.

[0040] This application does not limit the passthrough technology used, such as SR-IOV (Single Root I / O Virtualization, a virtualization technology) or PCIPassthrough (Peripheral Component Interconnect Passthrough), etc. SR-IOV can be used to enable physical devices such as network adapters and GPUs to directly provide virtual functions (VFs) to multiple virtual machines. Each virtual function can be configured independently and can directly access the host machine's hardware resources. PCI Passthrough can be used to pass physical devices such as network adapters and GPUs to secure containers, allowing the secure containers to directly access these physical devices, which can be called passthrough devices.

[0041] In some preferred embodiments, device passthrough can also be implemented based on VFIO (Virtual Function I / O) technology. VFIO is a module that enables passthrough for PCI devices on the host machine. PCI devices refer to external devices interconnected with the host machine's CPU via a PCI (Peripheral Component Interconnect) bus or a PCIe (Peripheral Component Interconnect express) bus. The computer device illustrated in Figure 1b is used as an example to illustrate passthrough from a target resource object to a secure container using VFIO. The difference between the computer device shown in Figure 1b and the computer device shown in Figure 1a is that the operating system 12 on the host machine includes not only KVM 121, but also VFIO 122 and drivers for various hardware resources on the host machine; VFIO 122 can directly pass the target resource object (e.g., GPU) to the secure container for use, which is beneficial to improving the access efficiency of the secure container to the GPU; in addition, in the computer device shown in Figure 1b, the application is illustrated as an AI framework and an AI application service developed and deployed based on the AI ​​framework. Among them, the user space 1110 of the client machine also includes the runtime library of the target resource object. In Figure 1b, the GPU runtime library is illustrated as an example.

[0042] The AI ​​framework can be PyTorch or Tensorflow, etc. PyTorch is an open source machine learning library for building and training deep learning models, and Tensorflow is an open source machine learning framework for building and training various types of machine learning models. The AI ​​application service can be inference service for large language models, image recognition service, natural language processing service, search advertising recommendation service, and function computing service, etc., which are not limited in this embodiment.

[0043] When the target resource object is implemented as a passthrough device of the secure container 111, the driver for the target resource object is located in the guest operating system 1112. Taking the target resource object as a GPU as an example, the guest operating system 1112 includes a GPU driver. Correspondingly, for non-target resource objects that are not implemented as passthrough devices of the secure container 111, their drivers are located in the host operating system 12. Of course, the host operating system 12 also includes virtualization-related functions such as KVM and VFIO.

[0044] Through the above-described secure container + passthrough device architecture, when the secure container interacts with the target resource object (such as a GPU), it can effectively reduce the overhead of the host operating system and improve I / O performance; at the same time, it can reduce data operations on the host machine.The number of replications between the system and the secure container reduces latency and increases throughput.

[0045] The following will describe in detail the method of state transition for the secure container under the above pass-through architecture, taking the interaction process between any two nodes as an example. For ease of description, the terms "source node" and "target node" will be used to describe any two nodes. The embodiments of this application do not limit the implementation form of the source node and the target node. Taking the host machine dimension as an example, the source node can be implemented as the host machine that needs to be state transitioned, and the target node can be implemented as another host machine; taking the GPU dimension as an example, the source node can be implemented as the GPU on the host machine that needs to be state transitioned, and the target node can be implemented as another GPU. The two GPUs can be located on the same host machine or on different host machines, and there is no limitation on this.

[0046] Taking the migration of workload in a first secure container on a source node to a second secure container on a target node as an example, the first secure container refers to the secure container on the source node that needs to undergo state migration. The workload in the first secure container mainly refers to the state data of the resource objects involved in the first secure container. Migrating the workload in the first secure container mainly refers to migrating the state of the resource objects included in the first secure container. The resource objects involved in the first secure container may include target resource objects managed by a driver and non-target resource objects not managed by a driver. The migration of workload in the first secure container includes the migration of the state data of the target resource objects in the first secure container and the migration of the state data of the non-target resource objects in the first secure container.

[0047] For example, taking a GPU as the target resource object, the workload in the first secure container may include GPU state data (e.g., the video memory list and its corresponding video memory content, other states in the GPU), CPU state data (e.g., CPU register states), storage and network state data (e.g., user temporary storage data, network addresses, disk states, etc.), memory state data (e.g., kernel data and user data of the secure container), and metadata of the secure container (e.g., management data of the secure container, information of the container engine layer, etc.). For ease of description, in some embodiments of this application, "memory state data" is simply referred to as "memory data". Among them, GPU state data belongs to the state data of the target resource object, while CPU state data, storage and network state data, memory data, and metadata of the secure container belong to the state data of non-target resource objects. Based on this, the state data of the target resource object (such as GPU state data) and the state data of non-target resource objects, such as CPU state data storage, network state data, memory data, and secure container state data, can be stored together.Metadata, etc., are migrated from the first secure container to the second secure container, thereby completing the migration process of the workload in the first secure container.

[0048] The migration of the workload in the first secure container to the second secure container can be completed when the first secure container is paused (i.e., cold migration) or when the first secure container is running (i.e., hot migration). In order to reduce the downtime during the workload migration process, this application provides a secure container migration method. From the perspective of the secure container, this migration method is a hot migration method, that is, the workload migration can be completed when the first secure container is running. Specifically, during the migration process, target resource objects and non-target resource objects are distinguished, different migration methods are adopted for different resource objects, and the migration processes of the two are merged. That is, hot migration is performed for non-target resource objects, and cold migration is performed for target resource objects during the hot migration process. In this way, during the cold migration of target resource objects, non-target resource objects can operate normally, which can optimize the downtime of target resource objects and non-target resource objects during the migration process, minimize the downtime of each resource object, and improve the user experience.

[0049] Further, in an optional embodiment of this application, a first virtual machine is set up in a first secure container, and a second virtual machine and a third virtual machine are set up in a second secure container; wherein, the first virtual machine is used to run the workload of the first secure container, and can serve as a data migration virtual machine for the first secure container, migrating the first state data generated by non-target resource objects and the state data of target resource objects generated by the first secure container to the second virtual machine. Correspondingly, the second virtual machine serves as an auxiliary virtual machine, and is used to receive state data from the first virtual machine and transfer this state data to the third virtual machine. In this embodiment, the auxiliary virtual machine can first take over part of the workload of the first secure container, so that the workload of the first secure container can be gradually migrated, avoiding the interruption of the workload of the first secure container during the migration process. On the other hand, since the auxiliary virtual machine undertakes part of the state data migration task, it reduces the load pressure on the third virtual machine. In addition, due to the existence of the second virtual machine, the migration of the first state data and the state data of the target resource object can be completed without starting the third virtual machine. Furthermore, the second virtual machine can be shut down after the migration of the state data of the target resource object is completed, ensuring that a virtual machine runs in the second secure container. This not only reduces the performance load of the target node, but also improves the isolation and security of the virtual machine on the target node and the correctness of the data.

[0050] Further, in an optional embodiment of this application, by adding virtual machines on the source node and the target node...The corresponding hot migration components, which cooperate with each other on different nodes, can effectively complete the rapid migration of workloads in the secure container, solving the problem of cloud vendors migrating secure containers with target resource objects.

[0051] The technical solutions provided by the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0052] As shown in FIG2a, it is a schematic diagram of the structure of a source node provided by an exemplary embodiment of this application. As shown in FIG2a, the source node 20 includes a hardware layer 211, which includes various hardware resources, such as CPU, GPU, network card, etc. Various software programs run on the hardware layer 211. These software programs include a host operating system 27 and a first container runtime component 23 and a first secure container 24 running on the host operating system 27. The first secure container 24 is created and managed by the first container runtime component 23, and the first container runtime component 23 and the first secure container 24 are located in the user space 25 of the host machine. The host operating system 27 includes a virtualization manager 26 and drivers for the corresponding hardware resources in the hardware layer 211 (not shown in FIG2a). In this embodiment, the first container runtime component 23 and the first secure container 24 running in the user space 25 of the host machine are used as an example for illustration, but it is not limited to this. Other programs can also run in the user space 25.

[0053] In this embodiment, the first secure container 24 can use various hardware resources in the hardware layer 211. These hardware resources are divided into target resource objects 28 and non-target resource objects 29, as shown in FIG2a. For a related introduction to target resource objects and non-target resource objects, please refer to the foregoing embodiments, which will not be repeated here. The first secure container 24 includes a guest operating system 22 and an application 21 running on the guest operating system 22. The application 21 is located in the user space 242 of the first secure container 24.

[0054] Accordingly, this embodiment also provides a target node. The implementation structure of the target node is the same as or similar to the implementation structure of the source node shown in FIG2a. It can be implemented with reference to the structure of the source node shown in FIG2a.

[0055] Further, the secure container migration method provided in this embodiment will be introduced with reference to the interactive flowchart of the secure container migration in FIG2b. As shown in Figure 2b, the migration of state data between the first secure container of the source node and the second secure container of the target node can be achieved based on the following steps:

[0056] Step S20: On the target node, a second secure container is pre-created.

[0057] Step S21: On the source node, during the operation of the first secure container, the first state data generated by the non-target resource objects of the first secure container is obtained;

[0058] Step S22: The first state data generated by the non-target resource objects of the first secure container is migrated to the second secure container pre-created on the target node;

[0059] Step S23: On the target node, save the first state data of the non-target resource object of the first security container into the non-target resource object of the second security container;

[0060] Step S24: On the source node, pause the operation of the target resource object of the first security container;

[0061] Step S25: Obtain the state data of the target resource object of the first security container;

[0062] Step S26: Migrate the state data generated by the target resource object of the first security container to the second security container on the target node;

[0063] Step S27: On the target node, save the state data of the target resource object of the first security container into the target resource object of the second security container;

[0064] Step S28: On the source node, pause the operation of the non-target resource object of the first security container;

[0065] Step S29: Obtain the second state data of the non-target resource object of the first security container, the second state data being the state data generated in the non-target resource object as described on page 8 / 35 of the specification, CN 122152423 A;

[0066] Step S210: Migrate the second state data of the non-target resource object in the first security container to the second security container on the target node;

[0067] Step S211: On the target node, store the second state data of the non-target resource object in the first security container in the non-target resource object of the second security container.

[0068] As shown in step S20, a second security container is pre-created on the target node to carry the workload in the first security container. Pre-creating the second security container on the target node means creating the second security container on the target node before hot-migrating the non-target resource object, providing a basis for hot-migrating the non-target resource object. As long as the creation of the second security container is completed on the target node before hot-migrating the non-target resource object, the specific creation time is not limited.

[0069] In this embodiment, the method for triggering the creation of the second security container on the target node is not limited. In an optional embodiment, the source node instructs the target node to create the second security container. For example, the source node instructs the target node to create the second security container before migrating the workload in the first security container. In another optional embodiment, the operator issues a migration command to the target node through the control node. Upon receiving the migration command, the target node creates a second secure container.

[0070] This application does not limit the specific implementation of creating the second secure container. In some exemplary embodiments, Kata Containers, etc., can be used to create the second secure container. Kata Containers is a lightweight virtualization technology that combines the convenience of containers with the security isolation features of virtual machines, and can be used to create a second secure container.Full container. Additionally, the second secure container can be created by the container runtime component on the source node instructing the target node to create it, or by the control node instructing the target node to create it; there is no limitation on this.

[0071] Wherein, in the process of multiple node interactions, the use of "first" and "second" to define the secure container is only used to distinguish between the secure container on the source node and the secure container on the target node, and does not limit the number, configuration, or type of the secure container. In subsequent embodiments, descriptions using "first" and "second" to define functional modules or components will also be seen, which are also only used to distinguish between the functional modules or components on the source node and the target node, and do not limit the number, priority, or type of functional modules or components, but rather descriptively distinguish functional modules or components based on the different requirements of the secure container migration task.

[0072] Similar to the first secure container, the second secure container includes target resource objects that depend on driver management and non-target resource objects that do not depend on driver management. It should be noted that the number and type of target resource objects contained in the second secure container need to be adapted to the number and type of target resource objects contained in the first secure container. Among them, quantity and type adaptation means that the target resource objects are of the same type, and the number of target resource objects contained in the second security container is greater than or equal to the number of target resource objects that need to be migrated in the first security container, so as to successfully migrate the state data of the target resource objects in the first security container to the target resource objects in the second security container. Correspondingly, the number and type of non-target resource objects contained in the second security container also need to be adapted to the number and type of non-target resource objects contained in the first security container, so as to successfully migrate the state data of non-target resource objects in the first security container to the non-target resource objects in the second security container. Regarding how to adapt between non-target resource objects, please refer to the adaptation content of target resource objects, which will not be repeated here.

[0073] In this embodiment, the target resource object refers to the resource object that the cloud vendor or security container cannot directly manage, but needs to rely on the driver for management / control. It is generally a hardware resource object that is directly connected to the security container, such as GPU, DPU or NPU, etc. Correspondingly, non-target resource objects refer to resource objects that can be directly managed by cloud vendors or security containers (see page 9 / 35 of the manual, CN 122152423 A). They do not require program-based management / control and can be hardware resource objects such as CPU, memory, or network cards; or virtual resource objects created through virtualization technology, such as Virtual Central Processing Units (vCPUs), virtual memory, or virtual network cards; or software resource objects.Source objects, such as the metadata of a Pod (container group) of a secure container, include engine information and node information. A Pod is the smallest scheduling unit in Kubernetes, which can be understood as a specific secure container cluster, containing one or more secure containers. Kubernetes, often abbreviated as k8s, is an open-source container orchestration platform used for automatically deploying, scaling, and operating containerized applications. Engine information typically refers to configuration information related to the Pod's runtime environment, while node information refers to metadata related to the node to which the Pod is scheduled.

[0074] In this embodiment, the migration of the workload for the first secure container mainly refers to the migration of the state data of the target resource object and non-target resource objects within the first secure container. The state data of the non-target resource object describes its working state and may include at least one of the following: currently stored data, storage space information, runtime space information, internal register information, and configuration information. Register information describes the current state of the registers, such as their value, ID (Identity document), or type. Configuration information describes various parameters and settings of the non-target resource object, such as CPU function switches, performance settings, or working modes. After obtaining the state data of the non-target resource object in the first security container, the state data of the non-target resource object in the first security container can be migrated to the non-target resource object in the second security container.

[0075] In this embodiment, hot migration is performed on the non-target resource object in the first security container to reduce the downtime of the non-target resource object. The hot migration process mainly refers to the fact that when the non-target resource object in the first security container is running, most of the state data can be migrated to the non-target resource object of the target node in the first round of migration. For ease of distinction and description, the state data migrated in this process is called the first state data; then, the operation of the non-target resource object is paused, and the remaining state data is migrated. The state data migrated in this process is called the second state data.

[0076] Based on the above, the hot migration process belongs to the segmented migration process, that is, the process of migrating the first state data when the non-target resource object is running and the process of migrating the second state data when the non-target resource object is paused. Among them, from the data type perspective, the state data generated by the non-target object mainly refers to the memory data in the first security container and related resource information, including but not limited to: CPU state data, storage and network state data, virtual machine state in the first security container, etc. The first state data includes most of the memory data and related resource information. Correspondingly, the second state data includes unmigrated memory data and related resource information.The state data includes the state data left over after the first state data migration and the state data newly generated by the non-target resource objects of the first security container after the first state data migration is completed and before the pause. The newly generated state data mainly refers to the state data generated during the cold migration of the state data of the target resource objects (including some memory data and related resource information). The second state data can be migrated after the state data migration of the target resource objects is completed.

[0077] The first state data and the second state data mainly include memory data. Taking memory data as an example, the migration process of the first state data includes: first, performing a full memory copy, and then copying the corresponding dirty page data in memory to the non-target resource objects of the target node multiple times according to the dirty page information, until the dirty page data in memory reaches the convergence state; where dirty page data refers to the modified memory pages. Correspondingly, taking memory data as an example, the migration process of the second state data includes: gradually copying the corresponding dirty page data in memory to the non-target resource objects of the target node according to the dirty page information, until all the dirty page data in memory is migrated.

[0078] The migration of the first state data refers to migrating the first state data to the non-target resource object of the second security container on the target node when the non-target resource object of the first security container on the source node is in a running state (i.e., without shutdown). As shown in Figure 2b, the migration process may include: S21, obtaining the first state data generated by the non-target resource object of the first security container on the source node; S22, migrating the first state data generated by the non-target resource object of the first security container to the second security container; and S23, storing the first state data of the non-target resource object of the first security container in the non-target resource object of the second security container on the target node.

[0079] Further, after the first state data migration is completed, the state data generated by the target resource object in the first security container is migrated to the target resource object in the second security container. The state data of the target resource object is used to describe the state of the target resource object. Optionally, if the target resource object of the first security container has video memory, its status data may include a video memory list, video memory content, and other status data. Other status data refers to status data other than the video memory list and its corresponding video memory content, such as at least one of register information, working status information, and configuration information within the target resource object. The video memory list refers to a list of video memory regions allocated to different applications on the target resource object. Register information describes the current state of registers, such as their value, ID, or type. Configuration information describes various parameters and settings of the target resource object, such as...The enabling / disabling of graphics card functions, performance settings, or display modes, etc. Working status information can be used to describe the working status of the target resource object, including at least one of the following: the ID of the target resource object, power status, task allocation, etc.

[0080] During the migration of the target resource object's status data, on the one hand, it depends on the normal operation of the first security container, and on the other hand, it requires the target resource object's status data in the first security container to be in a stable state, i.e., unchanged. Based on this, cold migration of the target resource object's status data can be performed. Cold migration refers to the migration of status data while the target resource object is in a paused state. For example, as shown in step S24 of Figure 2b, the operation of the target resource object in the first security container can be paused during the operation of the first security container. Optionally, the operation of the target resource object in the first security container can be paused through the driver of the target resource object. Then, as shown in step S25 of Figure 2b, the status data of the target resource object in the first security container is obtained while the target resource object is in a paused state, thus maintaining the consistency of the target resource object's status data. Furthermore, as shown in step S26 of Figure 2b, the state data generated by the target resource object in the first security container is migrated to the second security container; and as shown in step S27 of Figure 2b, the state data of the target resource object in the first security container is stored in the target resource object in the second security container.

[0081] During the migration of the state data of the target resource object, the non-target resource object in the first security container is running normally. Therefore, during this period, the non-target resource object may generate new state data, and when migrating the first state data, some dirty page data may remain and not be migrated. Based on this, after the cold migration process is completed, further, as shown in step S28 of Figure 2b, the non-target resource object can be paused so that the state data of the non-target resource object in the first security container is in a stable state, and then the second state data of the non-target resource object can be migrated. As shown in step S29 of Figure 2b, after pausing the operation of the non-target resource object of the first security container, the second state data can be obtained; further, as shown in step S210 of Figure 2b, the second state data generated by the non-target resource object of the first security container is migrated to the non-target resource object of the second security container; further, as shown in step S211 of Figure 2b, on the target node, the second state data of the non-target resource object of the first security container is stored in the non-target resource object of the second security container, so as to migrate the workload in the first security container to the second security container.

[0082] In the embodiments of this application, during the hot migration process for the first security container, the target resource object and the non-target resource object are distinguished, and different migration methods are adopted for the non-target resource object and the target resource object.The method integrates the migration processes of the target resource object and the hot migration process of the non-target resource object. In this way, during the cold migration of the target resource object, the non-target resource object can operate normally. This optimizes the downtime caused by the migration of different resource objects, minimizes the downtime of each resource object during the migration process, and improves the user experience. In this way, the rapid migration of workloads in secure containers can be completed, solving the problem of cloud vendors migrating secure containers with target resource objects. In addition, this method can better complete the rapid migration of workloads in secure containers, solving the problem of cloud vendors migrating secure containers with target resource objects.

[0083] The secure container migration method in the embodiments of this application can be applied to a variety of scenarios. For example, in application upgrade, resource optimization, or application failure scenarios, the workload of the secure container of the source node can be efficiently migrated to the secure container of the target node through the above method. Specifically, in the application upgrade scenario, the smooth transition between the old and new versions is very important. In resource optimization scenarios, migrating workloads can achieve a more even load distribution, avoid the formation of hotspot nodes, and improve the overall system stability and response speed. In application failure scenarios, when a node fails, services can be restored by migrating workloads to other unaffected nodes, ensuring data security.

[0084] In the embodiments of this application, each secure container includes a virtualization management module and a virtual machine. The virtualization management module works with the virtualization manager in the host operating system to perform various management functions of the virtual machine, including but not limited to: creation, suspension, startup, shutdown, etc. The virtualization management module in the secure container can be, but is not limited to: QEMU (Quick EMUlator, virtual operating system emulator), but is not limited to this. QEMU is an open-source virtual machine monitor and emulator that provides management and control capabilities for these virtual machines. QEMU is a virtualization management module located in the host user space. Furthermore, each secure container corresponds to a container runtime component. Besides creating and managing secure containers, the container runtime component can also act as a control node during the secure container migration process, responsible for the overall management and control of the migration process. However, this control node is not limited to the container runtime component; a dedicated control node for secure container migration can also be implemented on the source node. The steps in the embodiment shown in Figure 2b can be completed under the control of the control node.

[0085] Optionally, as shown in Figure 3a, one internal architecture of the source node 30a includes: a hardware layer 39a, a host operating system...The system 36a and the host machine's user space 35a are included. The hardware layer 39a includes various hardware resources. The first secure container 34a can use these hardware resources, which are divided into target resource objects 37a and non-target resource objects 38a. Figure 3a illustrates this using the example of target resource object 37a including the GPU and non-target resource objects 38a including the CPU and network card, but it is not limited to these examples. The host machine operating system 36a includes a first virtualization manager 361a and drivers for the non-target resource objects 38a (not shown in Figure 3a). The host machine's user space 35a includes a first container runtime component 33a and a first secure container 34a created by the first container runtime component 33a; the first secure container 34a includes a first virtual machine 31a and a first virtualization management module 32a; the first virtual machine 31a includes a guest operating system 315a and a user space 316a; the guest operating system 315a includes at least a driver 313a for the target resource object, and optionally, may also include a driver for an RDMA network card; the user space 316a includes at least an application 311a.

[0086] Optionally, as shown in FIG3b, one internal architecture of the target node 30b includes: a hardware layer 39b, a host operating system 36b, and a host machine's user space 35b. The hardware layer 39b includes various hardware resources. The second secure container 34b can use these hardware resources in the hardware layer 39a. These hardware resources are divided into target resource objects 37b and non-target resource objects 38b. In Figure 3b, the target resource object 37b includes a GPU, and the non-target resource objects 38b include a CPU and a network card, but the illustration is not limited to these examples. The host operating system 36b includes a second virtualization manager 361b and drivers for the non-target resource objects 38b (not shown in Figure 3b). The host machine's user space 35b includes a second container runtime component 33b and a second secure container 34b created by the second container runtime component 33b; the second secure container 34b includes a second virtual machine 31b, a third virtual machine 31c, and a second virtualization management module 32b; the second virtual machine 31b includes a guest operating system 315b and a user space 316b, and the third virtual machine 31c includes a guest operating system 315c and a user space 316c; the guest operating system 315b includes at least a driver 313b for the target resource object, and optionally, a driver for an RDMA network card; the guest operating system 315c includes at least a driver 313c for the target resource object, and optionally, a driver for an RDMA network card; the user space 316b includes at least an application...311b; User space 316c includes at least application 311c.

[0087] Wherein, the first virtualization management module 32a and the first virtualization manager 361a in the host operating system 36a of the source node 30a cooperate to perform various management of the first virtual machine. Wherein, the second virtualization management module 32b and the second virtualization manager 361b in the host operating system 36b of the target node 30b cooperate to perform various management of the second virtual machine and the third virtual machine. The target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the target resource object and non-target resource object of the second secure container are mounted on the second virtual machine and the third virtual machine, respectively, and the second virtual machine has access permissions to the non-target resource object mounted on the third virtual machine.

[0088] Based on the above, and in conjunction with the interactive flowchart of secure container migration in FIG3c, the secure container migration method provided in this embodiment will be introduced. As shown in Figure 3c, the migration of state data between the first secure container of the source node and the second secure container of the target node can be achieved based on the following steps:

[0089] S31: On the source node, the first state data generated by the non-target resource object of the first secure container is sent to the second virtual machine of the second secure container on the target node through the first virtual machine;

[0090] S32: On the target node, the second virtual machine of the second secure container saves the first state data to the non-target resource object mounted on the third virtual machine based on the access permissions to the non-target resource object mounted on the third virtual machine;

[0091] S33: On the source node, the target resource object of the first secure container is suspended from running through the driver of the target resource object;

[0092] S34: The state data generated by the target resource object of the first secure container is migrated to the target resource object of the second secure container mounted on the second virtual machine; correspondingly, the second virtual machine saves the state data of the target resource object of the first secure container to the target resource object of the second secure container mounted on the second virtual machine;

[0093] S35: The first virtual machine is suspended from running through the first virtualization management module, so as to suspend the running of the non-target resource object of the first secure container;

[0094] S36: Obtain the second state data of the non-target resource object through the first virtualization management module, and send the second state data of the non-target resource object to the second virtualization management module of the second security container on the target node;

[0095] S37: On the target node, write the second state data into the non-target resource object of the second security container mounted on the third virtual machine through the second virtualization management module;

[0096] S38: Remount the target resource object of the second security container mounted on the second virtual machine to the third virtual machine, so as to migrate the state data of the target resource object of the first security container to the target resource object of the second security container.

[0097] In this embodiment, the execution order of step S38 is not limited, and it can be executed at any time after step S34.

[0098] In the above embodiment, the execution object for migrating the first state data is not limited. For example, as shown in FIG3a, it can be a component added to the first container runtime component 33a, the first virtualization management module 32a, or the first virtual machine 31a. In an optional embodiment, a hot migration component is added to the first virtual machine 31a, and the first state data is migrated to a non-target resource object in the second secure container through the hot migration component. The implementation method of the hot migration component and its deployment location in the first virtual machine 31a are not limited. It can be implemented in the user space 316a of the first virtual machine, or it can be implemented in the guest operating system 315a, or it can be distributed between the user space of the first virtual machine and the guest operating system 315a of 316a. As shown in Figure 3a, the hot migration component includes a hot migration proxy component 312a located in the user space 316a of the first virtual machine (page 13 / 35 of the specification, 18 CN 122152423 A) and a hot migration driver component 314a located in the guest operating system 315a. With the cooperation of the hot migration proxy component 312a and the hot migration driver component 314a, the migration of the first state data can be completed.

[0099] As shown in Figure 3a, the first virtual machine 31a has target resource objects and non-target resource objects mounted on it. The first virtual machine 31a includes a first hot migration component (not shown in the figure). The first hot migration component is used at least to migrate the first state data of the non-target resource objects. In this embodiment, the specific implementation structure and location of the first hot migration component are not limited. In one example, as shown in FIG3a, the first hot migration component includes a hot migration agent component 312a located in the user space 316a of the first virtual machine and a hot migration driver component 314a located in the guest operating system 315a; wherein, the hot migration driver component 314a is at least used to obtain first state data, and the hot migration agent component 312a is at least used to send the first state data.

[0100] Further, a pre-created second security container 34b is running on the target node 30b, and the second security container 34b includes a second virtual machine 31b and a third virtual machine 31c. The target resource object and non-target resource object of the second security container 34b are respectively mounted on the second virtual machine 31b and the third virtual machine 31c. The second virtual machine 31b includes a second hot migration component (not shown in the figure), and the second virtual machine 31b has access permissions to the non-target resource object mounted on the third virtual machine. The second hot migration component is at least used to receive the first state data sent by the first hot migration component, and based on the access permissions of the third virtual machine 31c.Access permissions to non-target resource objects are granted, and the first state data is saved to the non-target resource object mounted on the third virtual machine 31c.

[0101] In this embodiment, the specific implementation structure and location of the second hot migration component are not limited. In one example, as shown in FIG3a, the second hot migration component includes a hot migration proxy component 312b located in the user space 316b of the second virtual machine and a hot migration driver component 314b located in the guest operating system 315b; wherein, the hot migration proxy component 312b is at least used to receive the first state data from the hot migration proxy component 312a, and the hot migration driver component 314b is at least used to save the first state data to the non-target resource object mounted on the third virtual machine. In this embodiment, during the operation of the first secure container, the hot migration proxy component 312a of the first hot migration component calls the hot migration driver component 314a of the first hot migration component to gradually obtain the first state data generated by the non-target resource object of the first secure container 34a, and the hot migration driver component 314a of the first hot migration component reports the obtained first state data to the hot migration proxy component 312a of the first hot migration component.

[0102] Optionally, taking the first state data including the memory data of the first security container 34a as an example, the hot migration drive component 314a of the first hot migration component is called to gradually send the first state data generated by the non-target resource objects of the first security container 34a, including: initially acquiring most of the memory data generated by the non-target resource objects of the first security container 34a, and copying the memory data of the non-target resource objects acquired initially to the memory of the target node 30b, and then acquiring the dirty page data in the memory according to the dirty page information and copying it to the memory of the target node 30b through each round of iterative copying, until the dirty page data reaches the convergence state, that is, exiting the iteration when the preset conditions are met. The memory data migrated in this process can be used as the first state data.

[0103] Further, as shown in step S31 of FIG3c, during the operation of the first security container, the first state data generated by the non-target resource objects of the first security container is sent to the second virtual machine through the first virtual machine. Optionally, the hot migration proxy component 312a of the first hot migration component sends the first state data to the hot migration proxy component 312b of the second hot migration component, so that the hot migration proxy component saves the first state data to a non-target resource object mounted on the third virtual machine.

[0104] Based on the above, as shown in step S32 of FIG3c, the second virtual machine saves the first state data to a non-target resource object mounted on the third virtual machine based on the access permissions of the non-target resource object mounted on the third virtual machine. Optional specification 14 / 35 pages 19 CN 122152423 AThe hot migration proxy component 312b of the second hot migration component calls the hot migration driver component 314b of the second hot migration component to write the first state data into the non-target resource object in the second security container.

[0105] Optionally, a third hot migration component can also be added in the third virtual machine to receive the first state data from the hot migration proxy component 312b. This embodiment does not limit the specific implementation structure and location of the third hot migration component. In one example, as shown in FIG3a, the third hot migration component includes a hot migration proxy component 312c located in the user space 316c of the third virtual machine and a hot migration driver component 314c located in the guest operating system 315c; wherein, the hot migration proxy component 312c is at least used to receive the first state data from the hot migration proxy component 312b, and the hot migration driver component 314c is at least used to save the first state data into the non-target resource object mounted on the third virtual machine. During this process, the hot migration proxy component 312b calls the hot migration driver component 314c to obtain the first state data stored in the second virtual machine, and sends the first state data to the hot migration proxy component 312c through the hot migration proxy component 312b. Upon receiving the first state data, the hot migration proxy component 312c saves the first state data to the non-target resource object mounted on the third virtual machine.

[0106] In this embodiment, the target resource object and non-target resource object of the second security container are respectively mounted on the second virtual machine and the third virtual machine; the second virtual machine is used to receive the first state data and the state data of the target resource object from the first security container, and transfer these state data to the target virtual machine, i.e., the third virtual machine, to complete the migration of these state data. Among them, the auxiliary virtual machine can first take over part of the workload of the first security container, so that the workload of the first security container can be gradually migrated, avoiding the interruption of the workload of the first security container during the migration process. On the other hand, since the auxiliary virtual machine undertakes part of the state data migration task, it reduces the load pressure on the target virtual machine.

[0107] Furthermore, due to the presence of the second virtual machine, the migration of the first state data and the state data of the target resource object can be completed without starting the third virtual machine; and, while completing the migration of the state data of the target resource object, the second virtual machine can be shut down, ensuring that only one virtual machine runs in the second secure container. This not only reduces the performance load on the target node but also improves the isolation, security, and data correctness of the virtual machine on the target node.

[0108] In this embodiment, the transmission method of the first state data of the non-target resource object is not limited. In some exemplary embodiments, the non-target resource object of the first secure container may include a regular network card. Based on this, the data can be transmitted via a regular network card using TCP / IP (Transmission Control Protocol).The Protocol / Internet Protocol (Transmission Control Protocol / Internet Protocol) copies the first state data generated by the non-target resource object of the first security container to the non-target resource object of the second security container. In some other exemplary embodiments, the non-target resource object of the first security container may include a Remote Direct Data Access (RDMA) network card. Based on this, the hot migration proxy component controlling the first hot migration component initiates an RDMA operation through the Remote Direct Data Access connection (RDMA connection) between the source node and the target node, and then copies the first state data generated by the non-target resource object of the first security container to the non-target resource object of the second security container through the RDMA operation. The RDMA network card can support RDMA technology, which is used to directly access memory between different nodes without the intervention of the CPU, thereby improving data transmission efficiency and reducing transmission latency. Therefore, based on the RDMA operation issued by the RDMA network card, the first state data of the non-target resource object of the first security container can be migrated to the non-target resource object of the second security container more efficiently.

[0109] Further, after the first state data migration is completed, the state data of the target resource object of the first security container is migrated. As shown in step S33 of Figure 3c, the operation of the target resource object of the first security container is paused. As shown in step S34 of Figure 3c, the state data generated by the target resource object of the first security container is cold-migrated to the target resource object of the second security container. Specification 15 / 35 pages 20 CN 122152423 A

[0110] In this embodiment, the execution object for cold migration of the state data of the target resource object is not limited. For example, it can be the first container runtime component, the first virtualization management module, or a component in the first virtual machine. In an optional embodiment, the component in the first virtual machine is used as an example. For example, it can be the driver of the target resource object, or a hot migration component added to the virtual machine. The process of cold migration of the state data of the target resource object based on the driver of the target resource object is described below.

[0111] In this embodiment, the operation of the target resource object of the first security container is paused through the driver of the target resource object. In this embodiment, the implementation method of "pausing the operation of the target resource object of the first security container through the driver of the target resource object" is not limited. In some optional embodiments, the guest operating system of the first secure container includes a driver for the target resource object, which is used to interact with user-space applications. The principle of the interaction between the target resource object driver and user-space applications is described below.

[0112] In this embodiment, the user space of the first secure container may include a runtime library for the target resource object.In this context, the runtime library of the target resource object is built on the driver of the target resource object. The runtime library encapsulates the functions provided by the driver and can serve as a bridge for the application to interact with the driver of the target resource object. In other words, the runtime library of the target resource object can be used for the application to interact with the driver of the target resource object. That is, the application can initiate various requests to the driver of the target resource object through the runtime library of the target resource object. When the application initiates a request through the runtime library of the target resource object, the request can be transformed into a call to the driver of the target resource object.

[0113] Taking the GPU as an example, the runtime library of the GPU can be CUDA (Compute Unified Device Architecture). CUDA can encapsulate the functions provided by the GPU. When the application sends a request through CUDA, the sent request can be transformed into a call to the driver of the GPU. That is, the application can use CUDA to call the driver of the GPU to interact with the GPU.

[0114] Based on this, the "pausing the operation of the target resource object of the first security container through the driver of the target resource object" in the aforementioned embodiment can be implemented based on the following implementation method one or implementation method two:

[0115] Implementation method one: intercepting the access request initiated by the application in the first security container using the interception function in the runtime library to pause the operation of the target resource object of the first security container. The access request is initiated by the application to the driver of the target resource object through the library function in the runtime library.

[0116] This embodiment does not limit the way of obtaining the interception function. In some exemplary embodiments, a function configuration interface can be provided to the user, and the interception function input by the user can be obtained in response to the user's configuration operation on the function configuration interface. Alternatively, a preset interception function can be obtained from the storage space corresponding to the first security container in a preset way. This embodiment does not impose any restrictions.

[0117] This embodiment can add the interception function in the runtime library in advance, but does not limit the specific implementation of adding the interception function. Optionally, the interception function can be added to the code of the runtime library based on the hook technology to intercept the function that needs to be intercepted in the runtime library. Among them, the hook technology is used to intercept the behavior of the function without modifying the original code.

[0118] Specifically, the target function to be intercepted can be selected from the library functions in the runtime library, where the library functions in the runtime library refer to the set of functions provided in the runtime library; after selecting the target function, a hooking method can be selected, such as the LD_PRELOAD method or the function replacement method, and the selected hooking method can be used to add the obtained intercepted function.Added to the runtime library code. Among them, the LD_PRELOAD method can load a custom dynamic link library by setting the LD_PRELOAD environment variable. This dynamic link library contains interception functions that replace the original library functions. Function replacement method specification 16 / 35 pages 21 CN 122152423 A The original library function can be replaced with an interception function by modifying the pointer of the library function.

[0119] The application in the first security container can initiate access requests to the driver of the target resource object through the library functions in the runtime library. Through the interception function in the runtime library, the above-mentioned driver access behavior of the application in the first security container can be effectively intercepted, so that the driver of the target resource object cannot be triggered by the application in the first security container, thereby achieving the purpose of suspending the operation of the target resource object in the first security container.

[0120] In this way, on the one hand, without relying on the driver, the purpose of suspending the operation of the target resource object in the first security container can be achieved by adding an interception function to the runtime library in the pre-stage. Since the interception function can be written according to the actual design requirements, this method has high controllability.

[0121] Implementation Method Two: Call the driver program of the target resource object to send a pause command to the target resource object of the first security container to pause the operation of the target resource object of the first security container. Specifically, the driver program can obtain the identification information of the target resource object, which is used to identify the identity of the target resource object. Then, the driver program can query the current status of the target resource object based on the identification information to ensure that it can be paused. Sending the pause command to the target resource object through the driver program can be implemented as a function call process, such as calling PauseDevice() or a similar operation. PauseDevice() is a function used to pause or stop the operation of a device. By calling the driver program in this way, the operation of the target resource object can be paused relatively quickly.

[0122] Through the above Implementation Method One or Implementation Method Two, the operation of the target resource object of the first security container can be paused relatively efficiently by adding an interception function in the runtime library or by calling the driver program.

[0123] After pausing the operation of the target resource object of the first security container, the status data of the target resource object of the first security container can be obtained through the driver program of the target resource object.

[0124] Further, in this embodiment, the state data is obtained through the driver program of the target resource object. The acquisition method can be implemented based on the following steps S41-S42:

[0125] Step S41: Call the target application interface in the driver program of the target resource object to obtain the first security scope.The target resource object's video memory list and its corresponding video memory content. The target application interface refers to the interface used to manage video memory usage. The video memory list refers to the list of video memory regions allocated to different applications on the target resource object, which may include information about the video memory segments used by each application, such as the starting address, size, type, and other relevant information of the video memory segment. The starting address of the video memory segment refers to the starting physical or virtual address of the video memory segment; the type of the video memory segment can be VRAM (a type of video memory segment) or WRAM (a type of video memory segment), etc. The video memory segment is used to store video memory content, which varies depending on the application running in the first secure container. In the case of running an AI framework and AI applications in the first secure container, the video memory content can be various data related to machine learning models, etc. Other relevant information may include the status of the video memory segment, such as idle, in use, etc., and may also include the priority of the video memory segment, which is not limited in this embodiment.

[0126] Optionally, before obtaining the video memory list of the target resource object in the first security container, the video memory list of the target resource object can also be determined based on the following method: First, a proxy function can be added to the runtime library to proxy access requests initiated by the application in the first security container to the driver of the target resource object through the library function in the runtime library. The proxy function can be added to the runtime library based on hook technology. The specific implementation method of this has been described in detail in the previous section "Adding an Interceptor Function to the Runtime Library," and will not be repeated here. In this way, when the application in the first security container initiates an access request to the driver of the target resource object, it can be initiated by the proxy function instead of directly through the library function in the runtime library. The proxy function can determine the request type of the access request. The request type may include memory allocation request, initialization request, and mode setting request, etc. In its specification, page 17 / 35, 22 CN 122152423 A, the memory allocation request may contain a list of video memory requested by the application from the target resource object. When the access request is a memory allocation request, the access request can be initiated and the list of video memory requested by the application can be read from it as the video memory list of the target resource object of the first security container.

[0127] Step S42: Call other application interfaces in the driver of the target resource object other than the target application interface to obtain other state data of the target resource object of the first security container. Other state data refers to state data other than the video memory list and its corresponding video memory content. There may be one or more other application interfaces, depending on the number of other state data types. One application interface can be used to obtain one or more other state data.According to this embodiment, no limitations are imposed. Other status data may include: internal register information, working status information, and configuration information.

[0128] The following will use other application programming interfaces implemented based on the NVML library as an example for illustration. The NVML library is a library for monitoring and managing target resource objects. Specifically, the API implemented based on the nvmlDeviceGetName() function in the driver of the target resource object can be called to obtain the name of the target resource object; the API implemented based on the nvmlDeviceGetUtilizationRates() function in the driver of the target resource object can also be called to obtain the runtime utilization rate of the target resource object. This embodiment does not impose any limitations.

[0129] In this way, the status data of the target resource object of the first security container can be obtained more accurately by calling the application programming interface of the driver of the target resource object. In this embodiment, the method of obtaining the status data of the target resource object of the first security container is not limited. In some exemplary embodiments, the status data of the target resource object can also be obtained by using the hot migration component on the first virtual machine and the second virtual machine, as in the above embodiments. This method has been described in the foregoing embodiments and will not be repeated here.

[0130] Further, the obtained state data of the target resource object of the first security container is sent to the second virtual machine of the second security container to save the state data in the target resource object of the second security container.

[0131] In this embodiment, the transmission method of the state data of the target resource object is not limited. In some exemplary embodiments, the target resource object of the first security container may include a regular network card. Based on this, the state data of the target resource object of the first security container can be copied to the target resource object of the second security container through the regular network card (based on TCP / IP). In some exemplary embodiments, the non-target resource object of the first security container may include a remote direct data access (RDMA) network card. Based on this, the hot migration proxy component of the first hot migration component is controlled to initiate an RDMA operation through the remote direct data access connection between the source node and the target node, and then the first state data generated by the non-target resource object of the first security container is copied to the non-target resource object of the second security container through the RDMA operation. Furthermore, when the target resource object is implemented as a GPU, the process of migrating the state data of the target resource object through the RDMA network card can be called GDR (GPU Direct RDMA). That is, the state data is directly transferred between the GPUs in the first and second secure containers through the RDMA network, without the need for memory copying through the CPU, thus enabling high-speed transmission of GPU state data.

[0132] Optionally, if the first hot migration component on the first virtual machine is used to migrate the state data of the target resource object, after pausing the operation of the target resource object, the hot migration proxy component in the first hot migration component calls the hot migration driver component to obtain the state data of the target resource object, and sends the obtained state data to the hot migration proxy component in the second hot migration component on the second virtual machine through the hot migration proxy component. The detailed implementation of the migration process of the state data of the target resource object through the cooperation of the first hot migration component and the second hot migration component is similar to the migration process of the first state data, and can be referred to the relevant content in the foregoing embodiments, which will not be repeated here.

[0133] In this embodiment, since the target resource object of the second security container is mounted in the second virtual machine, when the second virtual machine receives the state data of the target resource object on page 18 / 35 of the specification 23 CN 122152423 A, the target resource object bound to the second virtual machine can be rebound to the third virtual machine to migrate the state data of the target resource object of the first security container to the target resource object of the second security container. Specifically, the second secure container includes a second virtualization management module. The second virtualization management module works with the virtualization manager in the host operating system to manage the virtual machine in various ways. The second virtualization manager can be, but is not limited to, virtualization tools such as KVM. This embodiment does not impose any restrictions. By establishing a mapping relationship between the target resource object of the second secure container and the second virtual machine, the status data of the target resource object of the first secure container can be sent to the target resource object bound to the second virtual machine according to the mapping relationship. In this way, the target resource object bound to the second virtual machine in the second secure container can carry the status data of the target resource object of the first secure container.

[0134] Further, a mapping relationship is established between the target resource object of the second virtual machine and the third virtual machine. Further, based on the mapping relationship, the status data of the target resource object of the first secure container can be migrated to the target resource object bound to the second secure container and the third virtual machine. In addition, considering that both the second virtual machine and the third virtual machine run in the second secure container, after the status data of the target resource object has been migrated, the target resource object bound to the second virtual machine can be bound to the third virtual machine and the second virtual machine can be destroyed or suspended, and the third virtual machine can be run in the second secure container. Since the target resource object in the second security container is bound to the third virtual machine, it means that the state data of the target resource object in the first security container is migrated to the target resource object in the second security container that is bound to the second virtual machine.

[0135] In this way, on the one hand, the target resource object of the second security container and the second virtual machine can be established.The mapping relationship ensures the consistency of resources between the second virtual machine and the second secure container. That is, the second virtual machine can access the target resource object of the second secure container. On the other hand, when the state data of the target resource object of the first secure container is migrated, the second virtual machine can be destroyed to ensure that only one virtual machine runs in the second secure container, thereby improving the isolation and security of the virtual machine and the correctness of the data.

[0136] In addition to the above embodiments, the following optional implementation methods can also be used to save the state data of the target resource object of the first secure container in the target resource object of the second secure container:

[0137] In an optional implementation method, the target node includes two secure containers, namely a third secure container and a pre-created second secure container, the second secure container including the target resource object; first, the operation of the target resource object of the first secure container is paused, and the state data of the target resource object of the first secure container is saved in the target resource object of the second secure container; then, the target resource object of the second secure container is transferred to the third secure container to migrate the state data of the target resource object in the first secure container to the target resource object of the third secure container. Transferring the target resource object of the second security container to the third security container means remounting the target resource object mounted on the second security container to the third security container and removing the mounting relationship between the target resource object and the second security container. In this way, a new security container can be created on the target node, so that the state data of the target resource object of the first security container can be saved to the target resource object of the third security container more efficiently through the new security container.

[0138] Of course, in the case of the third security container, the first state data and the second state data can also be migrated to the non-target resource object in the second security container through the third security container. The specific migration process is similar to the above process and will not be repeated.

[0139] Further, after the migration of the state data of the target resource object is completed, as shown in step S35 of FIG3c, the operation of the non-target resource object of the first security container can be paused to ensure that the state data of the non-target resource object of the first security container is in a stable state, that is, unchanged. Furthermore, as shown in step S36 of Figure 3c, when the non-target resource object is in a paused state, the second state data of the non-target resource object in the first security container is obtained, thus maintaining the stability of the state data of the non-target resource object. Then, the second state data generated by the non-target resource object in the first security container is migrated to the non-target resource object in the second security container, so as to migrate the workload in the first security container to the second security container. The second state data at least includes the non-target resource object in the first security container...New state data generated during cold migration.

[0140] In the above embodiments, the execution object for migrating the second state data is not limited. For example, it can be a container runtime component or a virtualization management module. In an optional embodiment, taking the first virtualization management module 32a in the first secure container as an example, the first virtualization management module 32a can be used to pause the operation of the first virtual machine, thereby pausing the operation of the non-target resource object of the first secure container. At this time, the first virtual machine is shut down, so the relevant components in the first virtual machine cannot be used for workload migration. Therefore, in this embodiment, as shown in S36-S37 of FIG3c, the second state data is migrated to the non-target resource object of the second secure container through the cooperation between the first virtualization management module 32a and the second virtualization management module 32b. Herein, the first virtualization management module 32a or the second virtualization management module 32b refers to the module responsible for virtualization management located in the user space of the host machine, such as but not limited to VMM; further optionally, the first virtualization management module 32a or the second virtualization management module 32b can be implemented as QEMU, but is not limited to this.

[0141] In an optional embodiment, as shown in FIG3a, the first virtualization management module 32a includes a first migration management component 321a, and as shown in FIG3b, the second virtualization management module 32b includes a second migration management component 321b; then the second state data can be obtained through the first migration management component 321a and sent to the second migration management component 321b, so that the second migration management component 321b can write the second state data into the non-target resource object of the second security container.

[0142] In another optional embodiment, considering that the security container can manage non-target resource objects without relying on the driver of the non-target resource object, and can be managed through the container runtime component, therefore, unlike the previous method of pausing the operation of the target resource object in the first security container through the driver of the target resource object, for non-target resource objects, the operation of the non-target resource object in the first security container can be paused through the first container runtime component, so that the state data of the non-target resource object in the first security container is in a stable state, that is, unchanged. The first container runtime component can first obtain identification information such as the ID or name of the first secure container. Then, based on the identification information and the container pause command provided by the container runtime component, it can pause the operation of the non-target resource object of the first secure container. Next, the first container runtime component obtains the second state data and sends the second state data to the second container runtime component so that the second container runtime component can write the second state data into the non-target resource object of the second secure container.

[0143] In this embodiment, the present application does not refer to "obtaining through the first container runtime component corresponding to the first secure container, the first container runtime component obtains the second state data, the second container runtime component writes the second state data into the non-target resource object of the second secure container".The specific implementation of "second state data of non-target resource objects of the first secure container" is defined. In some exemplary embodiments, the first container runtime component can obtain and save the second state data of non-target resource objects of the first secure container based on the state saving tool in the user space of the first secure container. Alternatively, in some other exemplary embodiments, the user can manually obtain and save the second state data of non-target resource objects of the first secure container. For the first container runtime component, it can obtain and save the second state data of non-target resource objects of the first secure container in response to the user's save operation.

[0144] Wherein, the state saving tool of the first secure container is a secure container persistence tool, which is implemented based on checkpoint-restore technology. Checkpoint-restore technology is a technology for instant state saving and restoration of containers. This technology allows the container to save its runtime state data (called checkpoint) in one location and restore the state data (called restore) in another location. In short, the state saving tool can be used to save the runtime state of the secure container and can quickly restore it in subsequent processes. (20 / 35 pages of specification 25 CN 122152423 A) The runtime state is quickly restored. This embodiment does not limit the specific implementation of the state saving tool. In some exemplary embodiments, the state saving tool can be implemented as a CRIU (Checkpoint / Restore In Userspace) tool. The CRIU tool is a tool for implementing process state snapshots and restoration. It can be used to save the state data of resource objects in user space and restore the state data when needed.

[0145] Based on the above, the first container runtime component can call the state saving tool, which can obtain the second state data of non-target resource objects of the first security container. Specifically, the first container runtime component can call the state saving tool by sending a call instruction to the state saving tool. The state saving tool can take a state snapshot of the non-target resource objects of the first security container to obtain the corresponding second state data.

[0146] In this way, the second state data of non-target resource objects of the first security container can be obtained more efficiently based on the state saving tool.

[0147] Regardless of whether the execution object responsible for migrating the second state data is the first migration management component in the first virtualization management module or the first container runtime component, after obtaining the second state data of the non-target resource object of the first secure container, the second state data of the non-target resource object of the first secure container can be migrated to the non-target resource object of the second secure container.In some exemplary embodiments, the non-target resource object of the first security container may include a regular network interface card (NIC). Based on this, the second state data of the non-target resource object of the first security container can be copied to the non-target resource object of the second security container via TCP / IP using the regular NIC. Optionally, if the non-target resource objects of the first and second security containers include RDMA NICs, the first migration management component or the first container runtime component initiates an RDMA operation through a remote direct data access connection between the source node and the target node, and then copies the second state data generated by the non-target resource object of the first security container to the non-target resource object of the second security container through the RDMA operation. Since the RDMA operation can directly access memory between different nodes without CPU intervention, it is beneficial to improve data transmission efficiency and reduce transmission latency. Therefore, the second state data can be migrated to the non-target resource object of the second security container efficiently and quickly. In some optional embodiments, after the target node receives the status data of the target resource object and non-target resource object of the first secure container and saves the status data to the second secure container, that is, after completing the migration of the workload of the first secure container, it also includes: pausing the operation of the first secure container through the first container runtime component corresponding to the first secure container. The first container runtime component is used to create and manage the first secure container.

[0148] In other optional embodiments, the metadata of the first secure container can also be obtained through the first container runtime component corresponding to the first secure container. The metadata of the first secure container mainly refers to the relevant information of the running first secure container, such as the ID, configuration, name, status, etc. of the first secure container, but is not limited to this. Then, the metadata of the first secure container is synchronized to the second secure container so that the second secure container can restore the running state of the second secure container according to the metadata of the first secure container.

[0149] In this embodiment, the synchronization time of the metadata of the first secure container is not limited. For example, it can be before the first secure container performs the workload migration; or, it can be during the workload migration of the first secure container; or, it can be after the workload migration of the first secure container is completed.

[0150] Further, based on the status data of the first security container received by the second security container, the working state of the second security container is restored.

[0151] This application embodiment does not limit the specific implementation method of restoring the working state of the second security container. In some exemplary embodiments, the working state of the second security container can be restored based on the following steps S10-S13:

[0152] Step S10: Run the pre-created second security container.

[0153] Step S11: Restore the non-target resource object according to the status data in the non-target resource object of the second security container.Page 21 / 35 26 CN 122152423 A Working state of the image.

[0154] Step S12: Restore the working state of the target resource object according to the state data in the target resource object of the second security container.

[0155] In this embodiment, the process of restoring the working state of the second security container includes: receiving the metadata of the first security container and synchronizing it to the second security container through the container runtime component corresponding to the second security container; and / or running the second security container through the container runtime component corresponding to the second security container, and restoring the working state of the non-target resource object and the target resource object of the second security container according to the state data in the non-target resource object and the target resource object of the second security container respectively.

[0156] In this embodiment, the second security container can be restored to operation through the second container runtime component corresponding to the second security container in the target node, and then the working state of the non-target resource object and the target resource object of the second security container can be restored according to the state data in the non-target resource object and the target resource object of the second security container respectively. Among them, the working state of the target resource object can be restored through the driver of the target resource object. The driver for the target resource object is located in the guest operating system of the second secure container. The second container runtime component can call the driver for the target resource object to restore the working state of the target resource object through the driver.

[0157] In some optional embodiments, the user space of the second secure container may include a state recovery tool, which corresponds to the state saving tool in the user space and is also implemented based on checkpoint-restore technology. Based on the above, the second container runtime component can call the state recovery tool by sending a call instruction to the state recovery tool. The state recovery tool can provide the state data of the non-target resource object in the second secure container to the non-target resource object, so that the non-target resource object can restore its working state according to the state data. Alternatively, in some other exemplary embodiments, the user can manually restore the state data of the non-target resource object in the second secure container. For the container runtime component, it can respond to the user's restoration operation and restore the working state of the non-target resource object according to the state data of the non-target resource object in the second secure container.

[0158] In some optional embodiments, the second secure container may include a driver for the target resource object. The state data of the target resource object may include a video memory list and its corresponding video memory content and other state data, which will not be described in detail here. Based on this, the target application interface in the driver of the target resource object can be called to provide the video memory list and its corresponding video memory content in the target resource object of the second security container to the target resource object of the second security container.The target application programming interface (API) refers to the interface used to manage the usage of video memory. The details regarding the video memory list have been explained in detail above and will not be repeated here.

[0159] Subsequently, other application programming interfaces in the driver of the target resource object can be called to provide other state data of the target resource object in the second secure container to the target resource object of the second secure container, thereby restoring the working state of the target resource object. There can be one or more other application programming interfaces, depending on the number of different types of other state data. One application programming interface can be used to obtain one or more other state data; this embodiment does not impose any limitations.

[0160] In this way, the target node can efficiently restore the working state of the second secure container based on the received state data after completing the state data migration.

[0161] Optionally, before migrating the first secure container, the resource object can be pre-allocated and verified on the target node 30b through the second container runtime component 33b and the second virtualization management module 32b. During the resource object pre-allocation process, the metadata of the first container runtime component 33a in the first secure container 34a can be sent to the second container runtime component 33b in the second secure container 34b. This allows the second containerization component 33b to instruct the second virtualization management module 32b of the second secure container 34b to create resource objects adapted to the first secure container 34a based on the configuration information in the received metadata. This includes creating target resource objects and non-target resource objects belonging to the third virtual machine 31c. Furthermore, the second virtualization management module 32b also creates the second virtual machine 31b and the third virtual machine 31c based on the configuration information in the received metadata, and grants access permissions to the non-target resource objects belonging to the third virtual machine 31c to the second virtual machine 31b. This allows the second virtual machine 31b to write received state data into the non-target resource objects belonging to the third virtual machine 31c using this access permission. It should be noted that at this time, the third virtual machine 31c is created and the second virtual machine 31b is started. The startup timing of the third virtual machine 31c can be flexibly chosen, as long as it is before the state data migration corresponding to the resource object of the first secure container 34a is completed.

[0162] During the process of verifying the resources of the second secure container, it can be determined whether the second secure container has sufficient resources to carry the workload of the first secure container, or whether the second secure container is compatible with the first secure container. If the second secure container and the first secure container are compatible, the subsequent migration of the first secure container can be carried out. Among them, the second secure container being compatible with the first secure container includes: the target resource object contained in the second secure containerThe quantity and type need to be adapted to the quantity and type of target resource objects contained in the first security container; and the quantity and type of non-target resource objects contained in the second security container need to be adapted to the quantity and type of non-target resource objects contained in the first security container.

[0163] Based on the above embodiments, in addition to the source node and the target node, a management node may also be set in this embodiment to manage and control the source node and the target node. The management node can issue a container migration task to the source node, which is used to instruct the workload of the first security container on the source node to be migrated to the second security container on the target node. Optionally, the management node can respond to the task issuance event and issue a container migration task to the source node. The task issuance event can be implemented as detecting a fault in the source node, or as detecting a fault in the first security container on the source node, or as detecting that the workload of the first security container on the source node exceeds a preset range. The preset range can be set to any range according to actual design requirements, and this embodiment does not impose any restrictions.

[0164] In this embodiment, the control node can promptly issue a container migration task to the source node. The container migration task is used to instruct the migration of the workload of the first secure container on the source node to the second secure container on the target node.

[0165] Following the above embodiments, the secure container migration method provided in this application can be applied to intelligent computing service scenarios. The intelligent computing service can be deployed as an application in the user space of the first secure container in the above embodiments. The user can deploy the first secure container carrying the intelligent computing service on the source node through the control node, and directly connect the GPU on the source node to the first secure container as the computing power resource for the intelligent computing service. In this scenario, the intelligent computing service in the first secure container runs on the GPU. During the operation of the intelligent computing service, the following situations may occur: the user needs to upgrade the intelligent computing service, or the intelligent computing service malfunctions, or the source node malfunctions. In these cases, the intelligent computing service needs to be migrated, that is, the workload of the first secure container needs to be migrated. At this time, the control node can determine the target node to which it needs to be migrated and issue a state migration instruction to the source node to instruct the migration of the first secure container on the source node to the second secure container on the target node. Optionally, the control node can notify the target node to create a second secure container, or the source node can notify the target node to create a second secure container. The source node, according to the state transition instructions, executes the migration steps described in the above method embodiments to migrate the intelligent computing service from the source node's first secure container to the target node's second secure container. Then, based on the various state data migrated, the source node restores the states of the GPU, CPU, network, memory, etc., in the second secure container to ensure the GPU and CPU states remain unchanged before and after the migration.The CPU, storage, and network states remain consistent, allowing the intelligent computing service to resume operation in the second secure container, enabling it to continue providing services to customers.

[0166] The intelligent computing service scenario is only one application scenario of the secure container migration method provided in this application embodiment, and is not limited thereto. For example, the secure container migration method provided in this application embodiment can also be applied to the Function Compute service scenario, where the Function Compute service can be deployed as an application in the user space of the first secure container as described in the above embodiment. Function Compute provides an on-demand code execution method, where users only need to write and upload function code without worrying about the maintenance and expansion of the underlying infrastructure. The migration process of the secure container in this Function Compute application scenario is the same as in the above embodiment, and will not be repeated here.

[0167] As another example, the secure container migration method provided in this application embodiment can also be applied to the PAI (Platform for AI) scenario, where the AI ​​Platform service can be deployed as an application in the user space of the first secure container as described in the above embodiment. The AI ​​Platform service is used for target users to build, train, deploy, and manage artificial intelligence and machine learning models. The migration process of the secure container in this PAI application scenario is the same as in the above embodiments, and will not be repeated here.

[0168] This application embodiment also provides a secure container migration method for migrating the workload in a first secure container on a source node. The first secure container includes a target resource object and a non-target resource object. The target resource object is a pass-through device of the first secure container. This method is described from the perspective of the source node, as shown in Figure 4. The method includes:

[0169] Step S401: During the operation of the first secure container, the first state data generated by the non-target resource object of the first secure container is hot-migrated to the non-target resource object in the second secure container. The second secure container is a secure container pre-created on the target node.

[0170] Step S402: The operation of the target resource object of the first secure container is paused, and the state data generated by the target resource object of the first secure container is cold-migrated to the target resource object of the second secure container.

[0171] Step S403: The operation of the non-target resource object of the first secure container is paused, and the second state data generated by the non-target resource object of the first secure container is migrated to the non-target resource object of the second secure container.

[0172] In an optional embodiment, the target resource object and non-target resource objects of the first secure container are mounted on a first virtual machine of the first secure container, and the first virtual machine includes a first hot migration component; in the first secure container...During the full container operation, when hot-migrating the first state data generated by the non-target resource object of the first secure container to the non-target resource object in the second secure container, the process includes: during the operation of the first secure container, sending the first state data generated by the non-target resource object of the first secure container to the second hot-migrating component through the first hot-migrating component, so that the second hot-migrating component writes the first state data into the non-target resource object in the second secure container; wherein, the second hot-migrating component is a hot-migrating component in the second virtual machine of the second secure container.

[0173] In an optional embodiment, the first hot migration component includes: a hot migration agent component located in the user space of the first virtual machine, and a hot migration driver component located in the guest operating system of the first virtual machine; then, during the operation of the first secure container, when the first hot migration component sends the first state data generated by the non-target resource object of the first secure container to the second hot migration component so that the second hot migration component writes the first state data into the non-target resource object in the second secure container, the process includes: during the operation of the first secure container, calling the hot migration driver component to gradually obtain the first state data generated by the non-target resource object of the first secure container and reporting it to the hot migration agent component; controlling the hot migration agent component to send the first state data to the second hot migration component so that the second hot migration component writes the first state data into the non-target resource object in the second secure container.

[0174] In an optional embodiment, when calling the hot migration driver component to gradually obtain the first state data generated by the non-target resource object of the first secure container (pages 24 / 35 of the specification, 29 CN 122152423 A), it includes: calling the hot migration driver component to obtain the memory data generated by the non-target resource object of the first secure container as the first state data.

[0175] In an optional embodiment, the non-target resource object of the first secure container includes a remote direct data access network card; when controlling the hot migration proxy component to send the first state data to the second hot migration component, it includes: controlling the hot migration proxy component to send the first state data to the second hot migration component through the remote direct data access connection between the source node and the target node.

[0176] In an optional embodiment, the target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the first secure container further includes: a first virtualization management module, the first virtualization management module including: a first migration management component; when the non-target resource object of the first secure container is mounted on the first virtual machine of the first secure container, the first secure container further includes: a first virtualization management module, the first virtualization management module including: a first migration management component; when controlling the non-target resource object of the first secure container to send the first state data to the second hot migration component, it includes: a remote direct data access network card; when controlling the hot migration proxy component ...When the generated second state data is migrated to the non-target resource object of the second security container, it includes: obtaining the second state data through the first migration management component and sending the second state data to the second migration management component, so that the second migration management component can write the second state data into the non-target resource object of the second security container; wherein, the second migration management component is the migration management component in the second virtualization management module of the second security container.

[0177] In an optional embodiment, when pausing the operation of the non-target resource object of the first security container, it includes: pausing the operation of the first virtual machine through the first virtualization management module, so as to pausing the operation of the non-target resource object of the first security container.

[0178] In an optional embodiment, the target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container. The guest operating system of the first virtual machine includes: a driver for the target resource object; when pausing the operation of the target resource object of the first secure container and cold migrating the state data generated by the target resource object of the first secure container to the target resource object of the second secure container, the method includes: pausing the operation of the target resource object of the first secure container through the driver for the target resource object, and obtaining the state data of the target resource object of the first secure container; migrating the state data of the target resource object of the first secure container to the target resource object of the second secure container.

[0179] In an optional embodiment, the above method further includes: pausing the operation of the first secure container through the container runtime component corresponding to the first secure container, the container runtime component being used to create and manage the first secure container; and / or, obtaining the metadata of the first secure container through the container runtime component corresponding to the first secure container, and synchronizing the metadata of the first secure container to the second secure container.

[0180] Further optionally, the first secure container includes a guest operating system, the guest operating system includes a driver for the target resource object, and the user space of the first secure container includes a runtime library for the target resource object. Pausing the execution of the target resource object in the first secure container via the driver for the target resource object includes: intercepting access requests initiated by applications in the first secure container using interception functions in the runtime library to pause the execution of the target resource object in the first secure container; the access request is initiated by the application to the driver for the target resource object through library functions in the runtime library; or, calling the driver for the target resource object to send a pause command to the target resource object in the first secure container to pause the execution of the target resource object in the first secure container.

[0181] Further optionally, the target resource object of the first secure container is obtained through the driver for the target resource object.The state data of the target resource object includes: calling the target application interface in the driver of the target resource object to obtain the video memory list of the target resource object of the first secure container and its corresponding video memory content; calling other application interfaces in the driver of the target resource object other than the target application interface to obtain other state data of the target resource object of the first secure container, wherein the other state data refers to state data other than the video memory list and its corresponding video memory content.

[0182] Further optionally, the user space of the first secure container includes the runtime library of the target resource object, and the method further includes: using the proxy function in the runtime library to proxy the access request initiated by the application in the first secure container, and when the access request is a memory allocation request, obtaining the video memory list requested by the application as the video memory list of the target resource object of the first secure container; the access request is initiated by the application to the driver of the target resource object through the library function in the runtime library.

[0183] Further optionally, the method further includes: receiving a container migration task issued by the management node, wherein the container migration task is used to instruct the workload of the first secure container on the source node to be migrated to the second secure container on the target node.

[0184] The detailed implementation methods and beneficial effects of each step in the method of this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0185] This application embodiment also provides a secure container migration method for migrating the workload of a first secure container on a source node. A second secure container is pre-created on the target node. The second secure container includes a target resource object and a non-target resource object. The target resource object is a pass-through device of the second secure container. This method is described from the perspective of the target node. As shown in FIG5, the method includes:

[0186] Step 501: Receive first state data generated by the non-target resource object of the first secure container, and save the first state data of the non-target resource object of the first secure container in the non-target resource object of the second secure container.

[0187] Step 502: Receive the status data generated by the target resource object of the first secure container, and save the status data of the target resource object of the first secure container in the target resource object of the second secure container. The status data of the target resource object of the first secure container is transmitted when the target resource object of the first secure container is suspended.

[0188] Step 503: Receive the second status data generated by the non-target resource object of the first secure container, and save the second status data of the non-target resource object generated by the first secure container in the non-target resource object of the second secure container. The second status data includes at least the non-target resource object of the first secure container during the transmission of the target resource object of the first secure container.The state data generated during the state data of the resource object, and the second state data is transmitted when the target resource object of the first security container is suspended.

[0189] In an optional embodiment, the second security container includes: a second virtual machine and a third virtual machine, the target resource object and the non-target resource object of the second security container are respectively mounted on the second virtual machine and the third virtual machine, and the second virtual machine has access permissions to the non-target resource object mounted on the third virtual machine; when receiving the first state data generated by the non-target resource object of the first security container and saving the first state data in the non-target resource object of the second security container, it includes: the second virtual machine receives the first state data and saves the first state data to the non-target resource object mounted on the third virtual machine based on the access permissions.

[0190] In an optional embodiment, the second virtual machine includes a second hot migration component. When the second virtual machine receives the first state data and saves the first state data to a non-target resource object mounted on the third virtual machine based on the access permission, the method includes: receiving the first state data sent by the first hot migration component through the second hot migration component, and saving the first state data to a non-target resource object mounted on the third virtual machine based on the access permission; wherein, the first hot migration component is a hot migration component in the first virtual machine of the first secure container.

[0191] In an optional embodiment, the second hot migration component includes: a hot migration agent component located in the user space of the second virtual machine, and a hot migration driver component located in the guest operating system of the second virtual machine; Specification 26 / 35 pages 31 CN 122152423 A When receiving the first status data sent by the first hot migration component through the second hot migration component, and saving the first status data to a non-target resource object mounted on the third virtual machine based on the access permission, it includes: receiving the first status data sent by the first hot migration component through the hot migration agent component; and, based on the access permission, calling the hot migration driver component to save the first status data to a non-target resource object mounted on the third virtual machine.

[0192] In an optional embodiment, when saving the status data of the target resource object of the first secure container in the target resource object of the second secure container, it includes: saving the status data of the target resource object of the first secure container in the target resource object mounted on the second virtual machine; and remounting the target resource object mounted on the second virtual machine to the third virtual machine to migrate the status data of the target resource object of the first secure container to the target resource object of the second secure container.

[0193] In an optional embodiment, the second secure container further includes: a second virtualization management module, the second virtualization management module including: a second migration management component; when receiving second state data generated by the non-target resource object of the first secure container and storing the second state data in the non-target resource object of the second secure container, the method includes: using the second migration management component to receive the second state data sent by the first migration management component, and storing the second state data in the non-target resource object mounted on the third virtual machine; wherein, the first migration management component is the migration management component in the first virtualization management module of the first secure container.

[0194] In an optional embodiment, the above method further includes: receiving the metadata of the first secure container and synchronizing it to the second secure container through the container runtime component corresponding to the second secure container; and / or, running the second secure container through the container runtime component corresponding to the second secure container, and restoring the working state of the non-target resource object and the target resource object of the second secure container according to the state data in the non-target resource object and the target resource object of the second secure container, respectively.

[0195] Further optionally, the above method further includes: pre-creating a second security container on the target node, the second security container including a target resource object managed by a driver and a non-target resource object not managed by a driver.

[0196] Further optionally, the method further includes: running the second security container; restoring the working state of the non-target resource object according to the state data in the non-target resource object of the second security container; restoring the working state of the target resource object according to the state data in the target resource object of the second security container.

[0197] Further optionally, the user space of the second security container includes: a state restoration tool, which restores the working state of the non-target resource object according to the state data in the non-target resource object of the second security container, including: calling the state restoration tool to provide the state data in the non-target resource object of the second security container to the non-target resource object to restore the working state of the non-target resource object.

[0198] Further optionally, the second security container includes a driver for the target resource object, and the state data of the target resource object includes a video memory list and its corresponding video memory content and other state data; restoring the working state of the target resource object according to the state data in the target resource object of the second security container includes: calling the target application interface in the driver of the target resource object to provide the video memory list and its corresponding video memory content in the target resource object of the second security container to the target resource object of the second security container; calling other application interfaces in the driver of the target resource object other than the target application interface to provide other state data in the target resource object of the second security container to the target resource object.The target resource object of the second security container is given to restore the working state of the target resource object. The other state data refers to state data other than the video memory list and its corresponding video memory content.

[0199] Further optionally, the state data of the target resource object of the first security container is stored in the target resource object of the second security container specification page 27 / 35 32 CN 122152423 A, including: creating a second virtual machine in the second security container, binding the target resource object of the second security container to the second virtual machine, and migrating the state data of the target resource object of the first security container to the target resource object bound to the second virtual machine; creating a third virtual machine in the second security container, binding the target resource object of the second security container to the second virtual machine, and binding the target resource object bound to the second virtual machine to the third virtual machine, so as to migrate the state data of the target resource object of the first security container to the target resource object of the second security container.

[0200] The detailed implementation methods and beneficial effects of each step in the method of this embodiment have been described in detail in the foregoing embodiments, and will not be elaborated here.

[0201] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 501 to 503 can be the first device; or the execution subject of steps 501 and 502 can be the first device, and the execution subject of steps 502 and 503 can be the second device; and so on.

[0202] In addition, some processes described in the above embodiments and figures include multiple operations that appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear in this document or may be executed in parallel. The sequence numbers of the operations, such as 501, 502, etc., are only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of events, nor do they limit "first" and "second" to different types.

[0203] Figure 6 is a schematic diagram of a secure container migration device provided in another exemplary embodiment of this application, used to migrate workloads in a first secure container on a source node. The first secure container includes a target resource object and non-target resource objects, and the target resource object is a pass-through device of the first secure container. As shown in Figure 6, the device includes:

[0204] a first migration module 601, used to hot-migrate first state data generated by a non-target resource object of the first secure container to a non-target resource object in a second secure container during the operation of the first secure container.The second secure container is a secure container pre-created on the target node; the first pause module 602 is used to pause the operation of the target resource object of the first secure container and cold migrate the state data generated by the target resource object of the first secure container to the target resource object of the second secure container; the second pause module 603 is used to pause the operation of the non-target resource object of the first secure container and migrate the second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container.

[0205] In an optional embodiment, the target resource object and the non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the first virtual machine includes a first hot migration component; during the operation of the first secure container, when the first migration module 601 hot migrates the first state data generated by the non-target resource object of the first secure container to the non-target resource object in the second secure container, it is specifically used to: during the operation of the first secure container, send the first state data generated by the non-target resource object of the first secure container to the second hot migration component through the first hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container; wherein, the second hot migration component is a hot migration component in the second virtual machine of the second secure container.

[0206] In an optional embodiment, the first hot migration component includes: a hot migration agent component located in the user space of the first virtual machine, and a hot migration driver component located in the guest operating system of the first virtual machine; the first migration module 601, during the operation of the first secure container, gradually sends the first state data generated by the non-target resource object of the first secure container to the second hot migration component through the first hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container, specifically used for: during the operation of the first secure container, calling the hot migration driver component to obtain the first state data generated by the non-target resource object of the first secure container, and reporting it to the hot migration agent component; controlling the hot migration agent component to send the first state data to the second hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container.

[0207] In an optional embodiment, when the first migration module 601 calls the hot migration driver component to gradually obtain the first state data generated by the non-target resource object of the first security container, it is specifically used to: call the hot migration driver component to obtain the memory data generated by the non-target resource object of the first security container as the first state data.

[0208] In an optional embodiment, the non-target resource object of the first secure container includes a remote direct data access network card; when the first migration module 601 controls the hot migration proxy component to send the first state data to the second hot migration component, it is specifically used to: control the hot migration proxy component to send the first state data to the second hot migration component through the remote direct data access connection between the source node and the target node.

[0209] In an optional embodiment, the target resource object and the non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container, and the first secure container further includes: a first virtualization management module, the first virtualization management module including: a first migration management component; when the first pause module 602 migrates the second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container, it is specifically used to: obtain the second state data through the first migration management component and send the second state data to the second migration management component, so that the second migration management component can write the second state data into the non-target resource object of the second secure container; wherein, the second migration management component is the migration management component in the second virtualization management module of the second secure container.

[0210] In an optional embodiment, the second pause module 603, when pausing the runtime of the non-target resource object of the first secure container, is specifically used to: pause the operation of the first virtual machine through the first virtualization management module, so as to pause the runtime of the non-target resource object of the first secure container.

[0211] In an optional embodiment, the second pause module 603 is further used to: pause the operation of the first secure container through the container runtime component corresponding to the first secure container, the container runtime component being used to create and manage the first secure container; and / or, through the container runtime component corresponding to the first secure container, obtain the metadata of the first secure container, and synchronize the metadata of the first secure container to the second secure container.

[0212] FIG7 is a schematic diagram of another secure container migration device provided in another exemplary embodiment of this application, used to migrate the workload of the first secure container on the source node. The secure container migration device can be implemented in the target node, wherein a second secure container is pre-created on the target node, the second secure container including a target resource object and a non-target resource object, the target resource object being a pass-through device of the second secure container. As shown in Figure 7, the device includes:

[0213] a first storage module 702, configured to receive first state data generated by the target resource object of the first security container, and store the first state data of the target resource object of the first security container in the target resource object of the second security container.The second storage module 703 is used to receive the status data of the target resource object of the first security container and store the status data of the target resource object of the first security container in the target resource object of the second security container. The status data of the target resource object of the first security container is transmitted when the target resource object of the first security container is suspended. The third storage module 704 is used to receive the second status data generated by the non-target resource object of the first security container and store the second status data in the non-target resource object of the second security container. The second status data described on page 29 / 35 of the specification (CN 122152423 A) includes at least the status data generated by the non-target resource object of the first security container during the transmission of the status data of the target resource object of the first security container, and the second status data is transmitted when the target resource object of the first security container is suspended.

[0214] Further optionally, as shown in FIG7, the device further includes: a creation module 701, used to pre-create the second security container on the target node.

[0215] Further optionally, the second secure container includes: a second virtual machine and a third virtual machine, wherein the target resource object and the non-target resource object of the second secure container are respectively mounted on the second virtual machine and the third virtual machine, and the second virtual machine has access permissions to the non-target resource object mounted on the third virtual machine; when the first storage module 702 receives the first state data generated by the non-target resource object of the first secure container and stores the first state data in the non-target resource object of the second secure container, it is specifically used to: the second virtual machine receives the first state data and stores the first state data in the non-target resource object mounted on the third virtual machine based on the access permissions.

[0216] Further optionally, the second virtual machine includes a second hot migration component, and when the first storage module 702 receives the first state data and stores the first state data in the non-target resource object mounted on the third virtual machine based on the access permissions, it is specifically used to: receive the first state data sent by the first hot migration component through the second hot migration component and store the first state data in the non-target resource object mounted on the third virtual machine based on the access permissions; wherein, the first hot migration component is a hot migration component in the first virtual machine of the first secure container.

[0217] Further optionally, the second hot migration component includes: a hot migration agent component located in the user space of the second virtual machine, and a hot migration driver component located in the guest operating system of the second virtual machine; the first storage module 702 receives the first status data sent by the first hot migration component through the second hot migration component,When saving the first state data to a non-target resource object mounted on the third virtual machine based on the access permission, the second saving module 703 is specifically used to: receive the first state data sent by the first hot migration component through the hot migration proxy component; and, based on the access permission, call the hot migration driver component to save the first state data to a non-target resource object mounted on the third virtual machine.

[0218] Further optionally, when saving the state data of the target resource object of the first secure container in the target resource object of the second secure container, the second saving module 703 is specifically used to: save the state data of the target resource object of the first secure container in the target resource object mounted on the second virtual machine; and remount the target resource object mounted on the second virtual machine to the third virtual machine to migrate the state data of the target resource object of the first secure container to the target resource object of the second secure container.

[0219] Further optionally, the second secure container further includes: a second virtualization management module, the second virtualization management module including: a second migration management component; when the third storage module 704 receives the second state data generated by the non-target resource object of the first secure container and stores the second state data in the non-target resource object of the second secure container, it is specifically used to: receive the second state data sent by the first migration management component using the second migration management component, and store the second state data in the non-target resource object mounted on the third virtual machine; wherein, the first migration management component is the migration management component in the first virtualization management module of the first secure container.

[0220] Further optionally, the third storage module 704 is also used to: receive the metadata of the first secure container and synchronize it to the second secure container through the container runtime component corresponding to the second secure container; and / or, run the second secure container through the container runtime component corresponding to the second secure container, and restore the working state of the non-target resource object and the target resource object of the second secure container according to the state data in the non-target resource object and the target resource object of the second secure container respectively.

[0221] Further optionally, the creation module 701 is also used to: create a third secure container on the target node.

[0222] The internal functions and structure of the secure container migration device have been described above. As shown in FIG8, in practice, the secure container migration device can be implemented as a computer device, which can be implemented as the source node in the above embodiments, including: a memory 801, a processor 802, and a communication component 803.

[0223] The memory 801 is used to store computer programs and can be configured to store various other data to support computing.Operations on the computer device. Examples of this data include instructions for any application or method used to operate on the computer device, contact data, phonebook data, messages, pictures, videos, etc. Optionally, the memory 801 stores program code related to the first secure container and program code related to the container runtime component.

[0224] In some embodiments, the processor 802, coupled to the memory 801, is used to execute the computer program in the memory 801 (here, the computer program mainly refers to the program code related to the container runtime component on the source node) to: during the operation of the first secure container, hot-migrate the first state data generated by the non-target resource object of the first secure container to the non-target resource object in the second secure container, the second secure container being a secure container pre-created on the target node; suspend the operation of the target resource object of the first secure container and cold-migrate the state data generated by the target resource object of the first secure container to the target resource object of the second secure container; suspend the operation of the non-target resource object of the first secure container and migrate the second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container.

[0225] In an optional embodiment, the target resource object and non-target resource object of the first security container are mounted on the first virtual machine of the first security container, and the first virtual machine includes a first hot migration component; during the operation of the first security container, when the processor 802 hot migrates the first state data generated by the non-target resource object of the first security container to the non-target resource object in the second security container, it is specifically used to: during the operation of the first security container, send the first state data generated by the non-target resource object of the first security container to the second hot migration component through the first hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second security container; wherein, the second hot migration component is a hot migration component in the second virtual machine of the second security container.

[0226] In an optional embodiment, the first hot migration component includes: a hot migration agent component located in the user space of the first virtual machine, and a hot migration driver component located in the guest operating system of the first virtual machine; During the operation of the first secure container, the processor 802 gradually sends the first state data generated by the non-target resource object of the first secure container to the second hot migration component through the first hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container, specifically used for: During the operation of the first secure container, calling the hot migration driver component to obtain the non-target resource object of the first secure container.The processor 802 generates first state data from the target resource object and reports it to the hot migration proxy component; it controls the hot migration proxy component to send the first state data to the second hot migration component so that the second hot migration component writes the first state data into the non-target resource object in the second security container.

[0227] In an optional embodiment, when the processor 802 calls the hot migration driver component to gradually obtain the first state data generated by the non-target resource object of the first security container, it is specifically used to: call the hot migration driver component to obtain the memory data generated by the non-target resource object of the first security container as the first state data.

[0228] In an optional embodiment, the non-target resource object of the first security container includes a network card; when the processor 802 controls the hot migration proxy component to send the first state data to the second hot migration component, it is specifically used to: control the hot migration proxy component to send the first state data to the second hot migration component through the remote direct data access connection between the source node and the target node.

[0229] In an optional embodiment, the target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container. The first secure container further includes: a first virtualization management module, which includes: a first migration management component. When the processor 802 migrates the second state data generated by the non-target resource object of the first secure container to the non-target resource object of the second secure container, it is specifically used to: obtain the second state data through the first migration management component and send the second state data to the second migration management component so that the second migration management component can write the second state data into the non-target resource object of the second secure container; wherein, the second migration management component is a migration management component in the second virtualization management module of the second secure container.

[0230] In an optional embodiment, when the processor 802 suspends the operation of the non-target resource object of the first secure container, it is specifically used to: suspend the operation of the first virtual machine through the first virtualization management module to suspend the operation of the non-target resource object of the first secure container.

[0231] In an optional embodiment, the processor 802 is further configured to: pause the operation of the first secure container through the container runtime component corresponding to the first secure container, the container runtime component being used to create and manage the first secure container; and / or, obtain the metadata of the first secure container through the container runtime component corresponding to the first secure container, and synchronize the metadata of the first secure container to the second secure container.

[0232] In other embodiments, processor 802, coupled to memory 801, is configured to execute a computer program in memory 801 for: receiving first state data generated by a non-target resource object of the first security container and storing the first state data in a non-target resource object of the second security container; receiving state data generated by a target resource object of the first security container and storing the state data of the target resource object of the first security container in a target resource object of the second security container, wherein the state data of the target resource object of the first security container is transmitted while the target resource object of the first security container is suspended; and receiving second state data generated by a non-target resource object of the first security container and storing the second state data in a non-target resource object of the second security container; wherein the second state data includes at least state data generated by the non-target resource object of the first security container during the transmission of the state data of the target resource object of the first security container, and the second state data is transmitted while the target resource object of the first security container is suspended.

[0233] Further optionally, the second secure container includes: a second virtual machine and a third virtual machine, wherein the target resource object and the non-target resource object of the second secure container are respectively mounted on the second virtual machine and the third virtual machine, and the second virtual machine has access permissions to the non-target resource object mounted on the third virtual machine; when the processor 802 receives the first state data generated by the non-target resource object of the first secure container and saves the first state data in the non-target resource object of the second secure container, it is specifically configured to: the second virtual machine receives the first state data and saves the first state data to the non-target resource object mounted on the third virtual machine based on the access permissions.

[0234] Further optionally, the second virtual machine includes a second hot migration component. When the processor 802 receives the first state data in the second virtual machine and saves the first state data to a non-target resource object mounted on the third virtual machine based on the access permissions, it is specifically used to: receive the first state data sent by the first hot migration component through the second hot migration component, and save the first state data to a non-target resource object mounted on the third virtual machine based on the access permissions; wherein, the first hot migration component is a hot migration component in the first virtual machine of the first secure container.

[0235] Further optionally, the second hot migration component includes: a hot migration agent component located in the user space of the second virtual machine, and a hot migration driver component located in the guest operating system of the second virtual machine; processingWhen processor 802 receives the first status data sent by the first hot migration component through the second hot migration component and saves the first status data to a non-target resource object mounted on the third virtual machine based on the access permission, it is specifically configured to: receive the first status data sent by the first hot migration component through the hot migration proxy component; and, based on the access permission, call the hot migration driver component to save the first status data to a non-target resource object mounted on the third virtual machine.

[0236] Further optionally, when processor 802 saves the status data of the target resource object of the first secure container in the target resource object of the second secure container, it is specifically configured to: save the status data of the target resource object of the first secure container in the target resource object mounted on the second virtual machine; and remount the target resource object mounted on the second virtual machine to the third virtual machine to migrate the status data of the target resource object of the first secure container to the target resource object of the second secure container.

[0237] Further optionally, the second secure container further includes: a second virtualization management module, the second virtualization management module including: a second migration management component; when the processor 802 receives the second state data generated by the non-target resource object of the first secure container and saves the second state data in the non-target resource object of the second secure container, it is specifically used to: receive the second state data sent by the first migration management component using the second migration management component, and save the second state data in the non-target resource object mounted on the third virtual machine; wherein, the first migration management component is the migration management component in the first virtualization management module of the first secure container.

[0238] Further optionally, the processor 802 is also used to: receive the metadata of the first secure container and synchronize it to the second secure container through the container runtime component corresponding to the second secure container; and / or, run the second secure container through the container runtime component corresponding to the second secure container, and restore the working state of the non-target resource object and the target resource object of the second secure container according to the state data in the non-target resource object and the target resource object of the second secure container respectively.

[0239] Further, as shown in FIG8, the computer device also includes other components such as a display 804, a power supply component 805, and an audio component 806. FIG8 only schematically shows some components and does not mean that the computer device only includes the components shown in FIG8. In addition, the components within the dashed boxes in FIG8 are optional components, not mandatory components, and the specific components depend on the product form of the computer device. The computer device of this embodiment can be implemented as a desktop computer, a laptop computer, a smartphone, or an IoT device, etc.The device can also be a conventional server, cloud server, or server array, etc. If the computer device in this embodiment is implemented as a desktop computer, laptop computer, smartphone, etc., it may include the components within the dashed box in Figure 8; if the computer device in this embodiment is implemented as a conventional server, cloud server, or server array, it may not include the components within the dashed box in Figure 8.

[0240] This application embodiment also provides a cloud computing system, including: a source node and a target node; the source node and the target node cooperate with each other to migrate the workload in the first secure container on the source node to the second secure container on the target node; wherein, the process of cooperation between the source node and the target node and the working process of each node during the cooperation process can be referred to the foregoing embodiment, and will not be repeated here.

[0241] Correspondingly, this application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above method embodiments. Specification page 33 / 35 38 CN 122152423 A

[0242] Accordingly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps in the above method embodiments.

[0243] The above memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Disk or Optical Disk.

[0244] The above communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device housing the communication component can access wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes near-field communication (NFC).Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0245] The above-mentioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen may be implemented as a touch screen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touch, swipe, and gestures on the Touch Panel. The touch sensors may not only sense the boundaries of the touch or swipe action, but also detect the duration and pressure associated with the touch or swipe operation.

[0246] The above-mentioned power supply component provides power to various components of the device in which the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component is located.

[0247] The above-described audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals can be further stored in a memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0248] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, compact disc read-only memory (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0249] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of a flowchart and / or block diagram, as well as combinations of blocks in a flowchart and / or block diagram, can be implemented by computer program instructions. These computer programs are available.Instructions are sent to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce means for implementing the functions specified in the flowcharts or other programmable data processing apparatus (pages 34 / 35, CN 122152423 A).

[0250] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in the flowcharts or other programmable data processing apparatus and / or the block diagrams or other block diagrams.

[0251] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in the flowcharts or other programmable data processing apparatus and / or the block diagrams or other block diagrams.

[0252] In a typical configuration, a computing device includes one or more processors (Central Processing Units, CPUs), input / output interfaces, network interfaces, and memory.

[0253] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0254] Computer-readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage, or...Other magnetic storage devices or any other non-transfer media may be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0255] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0256] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application. Instruction manual, page 35 / 35, 40 CN 122152423 A, Figure 1a; Instruction manual, Figure 1 / 9, page 41 CN 122152423 A, Figure 1b; Instruction manual, Figure 2 / 9, page 42 CN 122152423 A, Figure 2a; Instruction manual, Figure 3 / 9, page 43 CN 122152423 A, Figure 2b; Instruction manual, Figure 4 / 9, page 44 CN 122152423 A, Figure 3a; Instruction manual, Figure 5 / 9, page 45 CN 122152423 A, Figure 3b; Instruction manual, Figure 6 / 9, page 46 CN 122152423 A, Figure 3c; Instruction manual, Figure 7 / 9, page 47 CN 122152423 A, Figure 5; Figure 6; Figure 7; Instruction manual, Figure 8 / 9, page 48 CN 122152423 A, Figure 8; Instruction manual, Figure 9 / 9, page 49 CN 122152423 A, Abstract Embodiments of the present application provide a secure container migration method, device, storage medium and program product. In the embodiments of the present application, in the process of hot migration forthe first secure container, the target resource object and the non-target resource object are distinguished, different migration modes are used for the non-target resource object and the target resource object, and the migration processes of the two are fused, that is, the cold migration process of the target resource object is fused in the hot migration process of the non-target resource object. In this way, in the cold migration process of the target resource object, the non-target resource object can normally operate, and the unavailable time of different resource objects due to migration can be optimized to minimize the unavailable time of each resource object in the migration process, thereby improving the user experience. In this way, the rapid migration of the workloads in the secure container can be completed, and the migration problem of the secure container with the target resource object by the cloud vendor is solved.

Claims

1. A secure container migration method, characterized in that, The method is used to migrate workloads in a first secure container on a source node, the first secure container including a target resource object and non-target resource objects, the target resource object being a pass-through device of the first secure container, the method comprising: During the operation of the first secure container, the first state data generated by the non-target resource object of the first secure container is hot-migrated to the non-target resource object in the second secure container, which is a secure container pre-created on the target node; Suspend the operation of the target resource object of the first security container, and cold migrate the state data generated by the target resource object of the first security container to the target resource object of the second security container; The operation of the non-target resource object of the first security container is suspended, and the second state data generated by the non-target resource object of the first security container is migrated to the non-target resource object of the second security container.

2. The method according to claim 1, characterized in that, The target resource object and non-target resource object of the first security container are mounted on the first virtual machine of the first security container, and the first virtual machine includes a first hot migration component; During the operation of the first secure container, the first state data generated by the non-target resource object in the first secure container is hot-migrated to the non-target resource object in the second secure container, including: During the operation of the first secure container, the first hot migration component sends the first state data generated by the non-target resource object of the first secure container to the second hot migration component, so that the second hot migration component writes the first state data into the non-target resource object in the second secure container; The second hot migration component is a hot migration component in the second virtual machine of the second security device.

3. The method according to claim 2, characterized in that, The first hot migration component includes: a hot migration agent component located in the user space of the first virtual machine, and a hot migration driver component located in the guest operating system of the first virtual machine; During the operation of the first secure container, the first hot migration component progressively sends the first state data generated by the non-target resource objects of the first secure container to the second hot migration component, so that the second hot migration component writes the first state data into the non-target resource objects in the second secure container, including: During the operation of the first secure container, the hot migration driver component is invoked to obtain the first state data generated by the non-target resource object of the first secure container, and the hot migration proxy component is reported. The hot migration proxy component is controlled to send the first state data to the second hot migration component, so that the second hot migration component writes the first state data into a non-target resource object in the second security container.

4. The method according to claim 3, characterized in that, The hot migration driver component is invoked to progressively acquire the first state data generated by the non-target resource object of the first security container, including: The hot migration driver component is invoked to obtain the memory data generated by the non-target resource object of the first security container, which is used as the first state data.

5. The method according to any one of claims 1-4, characterized in that, The target resource object and non-target resource object of the first secure container are mounted on the first virtual machine of the first secure container. The first secure container also includes: a first virtualization management module, which includes: a first migration management component. Migrating the second state data generated by the non-target resource object of the first security container to the non-target resource object of the second security container includes: The second status data is obtained through the first migration management component and sent to the second migration management component so that the second migration management component can write the second status data into the non-target resource object of the second security container; The second migration management component is the migration management component in the second virtualization management module of the second secure container.

6. The method according to any one of claims 1-4, characterized in that, Suspend the operation of non-target resource objects in the first security container, including: The first virtualization management module suspends the operation of the first virtual machine, thereby suspending the operation of non-target resource objects of the first security container.

7. The method according to any one of claims 1-4, characterized in that, The target resource object and non-target resource objects of the first secure container are mounted on the first virtual machine of the first secure container, and the guest operating system of the first virtual machine includes: the driver of the target resource object; Pausing the execution of the target resource object in the first security container and cold migrating the state data generated by the target resource object in the first security container to the target resource object in the second security container includes: The driver program for the target resource object is used to pause the operation of the target resource object in the first security container and obtain the status data of the target resource object in the first security container. The state data of the target resource object in the first security container is migrated to the target resource object in the second security container.

8. The method according to any one of claims 1-4, characterized in that, Also includes: The first secure container is paused by using the container runtime component corresponding to the first secure container. The container runtime component is used to create and manage the first secure container. and / or The metadata of the first secure container is obtained through the container runtime component corresponding to the first secure container, and the status data of the first secure container is synchronized to the second secure container.

9. A method for safely migrating a container, characterized in that, The method is used to migrate the workload of a first secure container on a source node to a pre-created second secure container on a target node. The second secure container includes a target resource object and a non-target resource object, wherein the target resource object is a pass-through device of the second secure container. Receive first state data generated by the non-target resource object of the first security container, and save the first state data in the non-target resource object of the second security container; Receives status data generated by the target resource object of the first secure container, and saves the status data of the target resource object of the first secure container in the target resource object of the second secure container. The status data of the target resource object of the first secure container is transmitted while the target resource object of the first secure container is suspended. Receive second state data generated by the non-target resource object of the first security container, and save the second state data in the non-target resource object of the second security container; The second state data includes at least the state data generated by the non-target resource object of the first security container during the transmission of the state data of the target resource object of the first security container, and the second state data is transmitted while the target resource object of the first security container is suspended.

10. The method according to claim 9, characterized in that, The second secure container includes a second virtual machine and a third virtual machine. The target resource object and the non-target resource object of the second secure container are respectively mounted on the second virtual machine and the third virtual machine, and the second virtual machine has access permissions to the non-target resource object mounted on the third virtual machine. Receiving first state data generated by a non-target resource object of the first security container and storing the first state data in a non-target resource object of the second security container includes: The second virtual machine receives the first status data and, based on the access permissions, saves the first status data to a non-target resource object mounted on the third virtual machine.

11. The method according to claim 10, characterized in that, The second virtual machine includes a second hot migration component. The second virtual machine receives the first state data and, based on the access permissions, saves the first state data to a non-target resource object mounted on the third virtual machine, including: The second hot migration component receives the first status data sent by the first hot migration component, and saves the first status data to a non-target resource object mounted on the third virtual machine based on the access permissions. The first hot migration component is the hot migration component in the first virtual machine of the first secure container.

12. The method according to claim 11, characterized in that, The second hot migration component includes: a hot migration agent component located in the user space of the second virtual machine, and a hot migration driver component located in the guest operating system of the second virtual machine; The second hot migration component receives the first status data sent by the first hot migration component, and saves the first status data to a non-target resource object mounted on the third virtual machine based on the access permissions, including: The hot migration proxy component receives the first status data sent by the first hot migration component; Based on the access permissions, the hot migration driver component is invoked to save the first state data to a non-target resource object mounted on the third virtual machine.

13. The method according to claim 10, characterized in that, Saving the state data of the target resource object of the first security container in the target resource object of the second security container includes: The state data of the target resource object of the first security container is stored in the target resource object mounted on the second virtual machine; The target resource object mounted on the second virtual machine is remounted to the third virtual machine to migrate the state data of the target resource object of the first security container to the target resource object of the second security container.

14. The method according to any one of claims 9-13, characterized in that, The second secure container further includes: a second virtualization management module, which includes: a second migration management component; Receiving second state data generated by a non-target resource object of the first security container, and storing the second state data in the non-target resource object of the second security container, including: The second migration management component receives the second status data sent by the first migration management component and stores the second status data in a non-target resource object mounted on the third virtual machine. The first migration management component is the migration management component in the first virtualization management module of the first secure container.

15. The method according to any one of claims 9-13, characterized in that, Also includes: The container runtime component corresponding to the second secure container receives the metadata of the first secure container and synchronizes it to the second secure container. And / or, The second secure container is run using the container runtime component corresponding to the second secure container, and the working states of the non-target resource object and the target resource object of the second secure container are restored according to the state data in the non-target resource object and the target resource object of the second secure container, respectively.

16. A computer device, characterized in that, It can be implemented as a source node, on which a first secure container is deployed, the first secure container including a target resource object and non-target resource objects; the target resource object is a pass-through device of the first secure container, wherein the first secure container is used to perform the steps of the method according to any one of claims 1-8; and / or, it can be implemented as a target node, on which a second secure container is deployed, the second secure container including a target resource object and non-target resource objects; the target resource object is a pass-through device of the second secure container, wherein the second secure container is used to perform the steps of the method according to any one of claims 9-15.

17. A computer-readable storage medium, characterized in that, When the computer program is executed by a processor, it causes the processor to perform the steps of the method according to any one of claims 1-8 and 9-15.

18. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, performs the steps of the method according to any one of claims 1-8 and 9-15.

19. A cloud computing system, characterized in that, include: A source node for performing the method of any one of claims 1-8, and a target node for performing the method of any one of claims 9-15; the source node and the target node cooperate with each other to migrate the workload in the first secure container on the source node to the second secure container on the target node.

Images