Systems and methods to enhance CPU isolation for critical applications

Abstract

A method for providing isolation from an operating system (OS) scheduler is provided. The method includes determining information about critical processes associated with an application to be assigned to isolated CPUs. The method includes determining a set of CPUs to be isolated. The method includes, after boot-time, performing one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

Description

SYSTEMS AND METHODS TO ENHANCE CPU ISOLATIONFOR CRITICAL APPLICATIONSTECHNICAL FIELD

[0001] Disclosed are embodiments related to systems and methods to enhance central processing units (CPU) isolation for critical applications.BACKGROUND

[0002] An operating system (OS) task scheduler typically treats all CPUs as available for scheduling process threads and preempts executing process threads giving CPU time to other applications. Boot-time Isolated CPUs, commonly known as “Isolcpus”, is the solution predominately adopted to isolate CPU cores to run critical applications. Booting the system with Isolcpus can prevent the OS scheduler from scheduling a process thread on the isolated CPU cores. Isolcpus can provide not only task- scheduling contention free CPUs, but can also keep system IRQs (interrupts tasks), system timers and other housekeeping tasks such as Read-Copy- Update (RCU), and kernel threads away from the CPU cores.

[0003] FIG. 1 illustrates, within a server 102, a critical application 106 provisioned on boot Isolcpus, while regular best-effort applications 104 use CPU cores 108 that are shared with other applications or tasks.SUMMARY

[0004] Even though Isolcpus provide the benefit of CPU isolation, there are some challenges associated in utilizing Isolcpus. For example, Isolcpus do not provide complete isolation for applications. Isolcpus are removed from the general scheduling domain. This prevents tasks (processes, threads) running in the system to be scheduled on the isolated CPUs. Tasks must be explicitly scheduled / migrated to Isolcpus using sched_setaffinityQ syscalls or using tools like taskset or numactl. But this does not prevent multiple applications getting scheduled on the same isolated CPU cores. Typically, critical applications will be scheduled and pinned on dedicated isolated cores to avoid interruption from other processes. Throughout this disclosure, wherever processes are referred to, it will also be understood to encompass threads, and tasks more generally, unless context demands a narrower understanding.

[0005] Multiple orchestrating entities managing and assigning CPUs to processes can schedule different processes on the same isolated cores. For example, a Kubernetes node agent Kubelet (which is responsible for managing containers scheduled on the node) using CPU Manager policies can schedule a container on certain Isolcpus at runtime. It is possible for other processes to be scheduled manually or through other orchestration mechanisms (e.g., Bash script, Openstack, Docker swarm) provisioning another process under its governance on the same Isolcpus where the critical application is running. This can lead to performance degradation of both processes in the system. Conflict can happen because competing orchestration mechanisms tend to maintain the CPU assignments locally within the orchestrator and do not share the CPU allocations across the different orchestrators. A simple solution would be to pre-define CPUs via policies for different orchestrating entities. But this is difficult in cloud scenarios where policy misconfigurations are common. Hence, it is necessary to provide better CPU isolation for critical applications against misconfigurations or malicious intents. An example described later in this disclosure shows an experimental demonstration highlighting the effect of degradation in application performance when multiple competing applications are scheduled on the same isolated CPU cores.

[0006] As another example, certain kernel processes can still run on Isolcpus. The Linux Scheduler prevents user-space processes and most of the kernel processes from getting scheduled on Isolcpus by controlling the CPU affinities for every process in the system. For each process in the system, the kernel maintains a list of CPUs a particular process is allowed to execute on. Typically, isolated cores are removed from each per-process allowed CPUs list. But certain kernel processes / threads still have isolated CPUs in their allowed CPUs list. This essentially means that certain kernel processes can be scheduled on the isolated cores where the critical process is already running and can steal CPU cycles from critical applications.

[0007] Isolated CPUs (Isolcpus) in the system under test are shown in the command prompt listing below. CPUs 30 and its hyper-thread sibling 134 are Isolcpus. root:-# cat / sys / devices / system / cpu / isolated30,134 root:-#root:-# cat / sys / devices / system / cpu / nohz_full30,134

[0008] The following command prompt listing shows the snip from a command that outputs all the processes in the system and their corresponding CPUs allowed list. Process IDs which do not have access to isolated CPUs include PID-2 and PID-10 (which cannot run on cores 30,134) and PID-23 (which can only run on core 1). Process IDs which still have access to isolated CPUs 30,134 (which are mostly kernel-space processes) include PID-3 to PID-6, PID-11 to PID- 14, PID- 16 to PID- 18, and PID-28. root:~ / projects / cpuirq-mt# for PID in 'Is / proc / I grep -E ' [0-9]+$’ I sort -nkl,l'; do cpus='grep -r . / proc / $PID / status I grep Cpus_allowed_list' ; echo PID-"$PID: $cpus”; donePID-1: Cpus_allowed_list: 0-207PID-2: Cpus_allowed_list: 0-29,31-133,135-207PID-3: Cpus_allowed_list: 0-207PID-4: Cpus_allowed_list: 0-207PID-5: Cpus_allowed_list: 0-207PID-6: Cpus_allowed_list: 0-207PID-8: Cpus_allowed_list: 0PID-10: Cpus_allowed_list: 0-29,31-133,135-207PID- 11 : Cpus_allowed_list: 0-207PID- 12: Cpus_allowed_list: 0-207PID- 13: Cpus_allowed_list: 0-207PID- 14: Cpus_allowed_list: 0-207PID- 15: Cpus_allowed_list: 0PID- 16: Cpus_allowed_list: 0-207PID-17: Cpus_allowed_list: 0-207PID-18: Cpus_allowed_list: 0-207PID-19: Cpus_allowed_list: 0PID-20: Cpus_allowed_list: 0PID-21: Cpus_allowed_list: 0PID-22: Cpus_allowed_list: 1PID-23 : Cpus_allowed_list: 1PID-24: Cpus_allowed_list: 1PID-25 : Cpus_allowed_list: 1PID-27: Cpus_allowed_list: 1PID-28: Cpus_allowed_list: 0-207PID-29: Cpus_allowed_list: 2PID-30: Cpus_allowed_list: 2PID-31: Cpus_allowed_list: 2PID-32: Cpus_allowed_list: 2PID-34: Cpus_allowed_list: 2

[0009] Hence it is necessary to provide better CPU isolation for critical workloads running on Isolcpus against misconfigurations and kernel processes. Prior works typically try to utilize boot-time allocated isolated CPUs (Isolcpus) and to provision critical workloads on the Isolcpus, without trying to provide either (i) enhanced isolation for critical workloads to ensure deterministic performance or (ii) security protection against denial-of-CPUs for critical applications, or preventing the execution of untrusted workloads on same CPU, and so on.

[0010] Advantages of proposed embodiments include at least the following. Embodiments provide more CPU isolation to critical applications than vanilla boot-timeIsolcpus, e.g., by further isolating them from processes running on root control groups (Cgroups), especially from the kernel processes. No changes are required to the applications, system components, and operating system. Embodiments can be seamlessly integrated into the existing systems. Embodiments provide for Reduced OS jitter for critical applications, improving characteristics for deterministic performance. Embodiments improve security, as no other processes could be pinned to the isolated cores.

[0011] According to a first aspect, a method for providing isolation from an operating system (OS) scheduler is provided. The method includes determining information about critical processes associated with an application to be assigned to isolated CPUs. The method includes determining a set of CPUs to be isolated. The method includes, after boot-time, performing one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

[0012] According to a second aspect, a server is provided. The server includes processing circuitry; and a memory. The memory contains instructions executable by the processing circuitry, whereby when executed the processing circuitry is configured to determine information about critical processes associated with an application to be assigned to isolated CPUs. The processing circuitry is configured to determine a set of CPUs to be isolated. The processing circuitry is configured to, after boot-time, perform one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

[0013] According to a third aspect, a computer program is provided, comprising instructions which when executed by the processing circuitry of a node cause the node to perform the method of any of the embodiments of the first aspect.

[0014] According to a fourth aspect, a carrier is provided, containing the computer program of the third aspect. The carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0016] FIG. 1 illustrates a server with different processes on boot isolated CPUs and shared CPUs.

[0017] FIG. 2 illustrates a server with CPU Isolation Enhancer according to some embodiments.

[0018] FIG. 3 illustrates a high-level flow sequence for CPU Isolation Enhancer according to some embodiments.

[0019] FIG. 4 illustrates a control group hierarchy according to some embodiments.

[0020] FIG. 5 illustrates a control group hierarchy according to some embodiments.

[0021] FIG. 6 illustrates a control group hierarchy according to some embodiments.

[0022] FIG. 7 illustrates a flowchart of CPU Isolation Enhancer according to some embodiments.

[0023] FIG. 8 illustrates a flowchart of CPU Isolation Enhancer according to some embodiments.

[0024] FIG. 9 illustrates a flowchart of CPU Isolation Enhancer according to some embodiments.

[0025] FIG. 10 illustrates the low CPU jitter experienced by test-app-instance-1 on CPUs 30 and 134.

[0026] FIG. 11 illustrates a spike in the CPU jitter experienced by “test-app-instance-1” when “test-app-instance-2” is co-located on the CPUs.

[0027] FIG. 12 illustrates a flowchart according to some embodiments.

[0028] FIG. 13 is a block diagram of an apparatus according to some embodiments.DETAILED DESCRIPTION

[0029] Embodiments provide a system and a method to provide enhanced CPU isolation at runtime, and on demand, for critical applications while being provisioned on Isolcpus. This may include traditional boot-time Isolcpus, or even dynamically provided Isolcpus, such as disclosed in a co-pending application being filed concurrently herewith.

[0030] Embodiments include a new software module located, for example, on each server of a cloud infrastructure and provided with access to a system’s CPU resources and control interfaces. Users and system administrators may provide information on the critical workloads, e.g., on the Isolcpus allocated to each critical application, the path to application Cgroup directory, and so on. In embodiments, the proposed software module may control the Cgroup hierarchy and the cpuset available to each Cgroup, thereby isolating CPU cores allocated to the critical processes from most of the other processes in the system. The software module may use kernel interfaces to control CPUs allowed for certain kernel processes (which cannot be controlled through using the Cgroup management method just mentioned), thereby providing complete CPU isolation to the critical process. When the critical application run-to-completion or is killed, the proposed module may dynamically revert the changes made to enhance the isolation of the critical application.

[0031] Embodiments provide for controlling (e.g., all) Cgroups to prevent other processes from running on the isolated CPU cores. Embodiments provide for migrating tasks from the root Cgroup to a non-root Cgroup and controlling the available CPUs for the assigned tasks. Embodiments provide for controlling non-migratable kernel processes running on the root Cgroup using a CPU Isolation Enhancer kernel-component and controlling the CPU affinities of the kernel processes and preventing them from getting scheduled on isolated cores.

[0032] FIG. 2 illustrates a system according to an embodiment. As shown, a server 202 may include a CPU Isolation Enhancer 204, which is a software module for improving the isolation of CPUs on demand, and which may leverage existing host OS low-level resource interfaces and / or resources. CPU Isolation Enhancer 204 may include a kernel threads CPU controller 206 that controls CPUs available for each kernel process / thread. CPU isolationEnhancer 204 may also include a Cgroups controller 208 that controls Cgroups hierarchy and CPUs available to Cgroups. CPU Isolation Enhancer 204 may take as input, for example, details of the application to be isolated and a list of Isolcpus allocated to the application.

[0033] Achieving better CPU isolation comprises, in some embodiments, modifying various parameters at one or both of an operating system’s user- space and kernel-space. CPU Isolation Enhancer 204 may be realized as two components, a Cgroup controller 208 running at user-space and Kernel threads CPU controller 206 provisioned in kernel-space (e.g., as a loadable kernel module).

[0034] CPU Isolation Enhancer 206 may take application information as input. For example, the application information input may include information about the critical application that needs to be shielded. This may include Process IDs (PIDs) and Thread IDs (TIDs) belonging to the application. A PID of the application may be used for managing and reorganizing Cgroups. Other non-limiting examples could be the application name, application Cgroup path, and so on, from which the PID of the application may be derived.

[0035] CPU Isolation Enhancer 206 may also take Isolcpus information as input. For example, Isolcpus information input may include a list of Isolcpus allocated to the application. Based on the user’s preference, the list could include all Isolcpus or a subset of Isolcpus for which the additional isolation is requested. In certain scenarios, the list of Isolcpus can be optional. CPUs allocated for the critical application can be fetched through Cgroups or through / proc interfaces.

[0036] FIG. 3 illustrates a high-level flow sequence of CPU Isolation Enhancer 204. An example of a more detailed flow sequence of each sub-component is shown in FIGS. 7-9.

[0037] CPU Isolation Enhancer 204 may be used to perform a method for enhancing CPU isolation for critical applications. This method may include one or more of the following steps to dynamically create enhanced CPU isolation for an application:

[0038] 1. A user, system administrator, and / or another entity present in the system (e.g.,Kubelet, Container runtimes, hypervisors, and so on) provides information about the critical tasks, including processes and threads (PIDs and TIDs) to be isolated and possibly theapplication’s current Cgroup information. CPU Isolation Enhancer 204 accepts this information as input.

[0039] 2. CPUs to be isolated may be explicitly provided by the user or otherwise determined, e.g., by reading / proc / < PID > / status file and parsing the Cpus_allowed_list parameter to identify the CPUs allocated for the critical application. CPU Isolation Enhancer 204 accepts this information as input.

[0040] 3. Controlling Cgroups (Domains) using Cgroup Controller module 208. To address the challenge mentioned above, where multiple applications can be scheduled on the same Isolcpus, it is necessary to prohibit non-critical tasks (processes and threads) from accessing isolated cores. One possible way to do this is by controlling cpusets in Cgroups in Linux or Job objects in Windows (generically referred to as Cgroups herein). Typically, Cgroups are mounted in / sys / fs / cgroup in Linux. Each Cgroups (folder) can inherit CPUs (or a subset of CPUs) from the parent Cgroup. CPUs available to the Cgroup are controlled via a cpuset. cpus file. To control the processes (PIDs) which can be part of a particular Cgroup, tasks (for CgroupsVl) or cgroup. procs (for CgroupsV2) can be utilized. Tasks (processes and threads) belonging to a Cgroup have access only to the CPUs allocated to the Cgroup. This provides an opportunity to control the available CPUs for a particular Cgroup, by which CPUs can be isolated and made available only to certain tasks - providing isolation from other tasks. FIG. 4 illustrates a Cgroups tree hierarchy in Linux.

[0041] 3-1. Managing Cpusets assigned to the Cgroups. Applications provisioned using cloud-native methods like Docker or Kubernetes create and follow a pre-defined hierarchy of folders under Cgroups. Each application has a dedicated Cgroup folder associated with it. For Example: docker.slice, kuberenetes. slice, and so on. If the application is allocated on the root Cgroup directly, then Cgroups Controller 208 may create a new folder for the application below the Cgroup’s root folder. The candidate CPUs allocated for the critical application may be removed from other Cgroups (that is, from Cgroups not in the path of the critical application). This will remove the permission for other tasks (processes, threads) to run on the candidate cores. Cgroups Controller 208 may perform a Depth-First-Search (DFS) post-order traversal and remove the candidate CPUs from each Cgroup (cpuset. cpus) not associated with the criticalapplication from bottom to top of the Cgroups tree hierarchy. Cgroups Controller 208 may also append the candidate CPUs only on the path that leads to the Cgroup where the critical application tasks will be running.

[0042] FIG. 5 illustrates a modified Cgroups after assigning candidate CPUs to the critical application (which, in this example, runs on “Isolated Container-A”). In this example, the application requests for two CPUs, hence core number “30” and its sibling hyper-thread “134” are assigned to the container-A. It is to be noted that, expect for the path that leads to the container-A, cores 30 and 134 are excluded from all other Cgroups dynamically. This provides isolation - other tasks (processes, threads) do not have access to CPU cores 30 and 134.

[0043] These steps help to isolate user-space and some system tasks on-demand from the CPUs on which the critical application will be provisioned. Additional steps may be used to reach full isolation.

[0044] 3-2. Migrating tasks from root Cgroup. In Cgroups, typically, tasks (PIDs orTIDs) are placed on a leaf Cgroup node. But this is not a requirement. One such scenario is where some tasks are assigned to the root Cgroup. The challenge is that the available cpuset in the root Cgroups cannot be modified. This means that all the CPUs in the system are part of the root Cgroup. Since the root Cgroup is the parent node of all other Cgroups, Linux poses a constraint from modifying the CPUs available to the root Cgroup. In summary, the tasks assigned to the root Cgroup still have access to all CPUs in the system, which includes the isolated CPUs 30 and 134 in this example. Misconfigured or malicious tasks can still be explicitly assigned to isolated cores.

[0045] FIG. 6 illustrates a periodic migration of tasks from the root Cgroup to a new Cgroup. A possible solution to the problem just identified is to create a new Cgroup under the root node and (e.g., periodically) migrate the tasks from the root Cgroup to the new Cgroup. Creating a new Cgroup allows the possibility to modify the cpuset (CPUs available) to the new Cgroup, since the constraint is only for the root Cgroup. In the proposed solution Cgroups Controller 208 may create a new Cgroup. It assigns the new Cgroup with all the CPUs except the Isolcpus assigned to the critical application and migrates the tasks from the root Cgroup to the new Cgroup as shown in FIG. 6. This migration may be performed periodically. For example,Cgroups Controller 208 may periodically check if there are tasks assigned to the root Cgroup and if so, it may migrate the tasks to the new Cgroup. An interval for this periodic check may be a configurable parameter and can be configured by system administrator.

[0046] FIG. 7 illustrates a flowchart for the Cgroups Controller 208 in order to isolate a critical application from other tasks according to some embodiments. As shown, Cgroups controller 208 operates within the CPU Isolation Enhancer 204. Application details (e.g., PID, Cgroup path) and a list of Isolcpus allocated to the application may be input into the Cgroups controller 208. At 702, Cgroups controller 208 checks to see if a Cgroup folder exists. If not, at 704, a new Cgroups folder below the root is created. Operation continues to 706, where Cgroups controller 208 manages cpusets assigned to the Cgroups. This management may include performing a depth-first search post-order traversal to remove Isolcpus allocated to the critical application from Cgroups not associated with the critical application. This management may also include performing a depth-first search pre-order traversal on the path leading to the critical application’s Cgroup to append the Isolcpus to the Cgroups along the path. At 708, Cgroups controller 208 checks whether this operation was successful. If not, an error is returned, otherwise, operation continues to 710 where tasks from the root Cgroup are migrated away from the root Cgroup. This migration may include creating a new Cgroup below the root Cgroup, assigning CPUs to the new Cgroup except for the Isolcpus assigned to the critical application, and periodically migrating tasks from the root Cgroup to the new Cgroup. An interval for the periodic check may be provided here as an input. The success or error status of the operation may then be returned.

[0047] 3-3. Controlling CPU affinity for Kernel Processes at the root Cgroup usingKernel threads CPU Controller 206. Even after migrating processes from the root Cgroup, there may still be kernel processes that cannot be migrated out of the root Cgroup (“non-migratable processes”). This is due to a limitation posed by Linux kernel for safety reasons. For example, the Linux community has had discussion about and provided a patch for preventing kthreadd processes (typically the PID-2) from migrating out of root Cgroup. That is, there is code in the Linux kernel that prevents the task from migration and prints a corresponding error to the user. Similarly, certain other kernel tasks are prevented from migrating to non-root Cgroups.

[0048] Now the challenge is that certain kernel processes (“non-migratable processes”) cannot be migrated to a non-root Cgroup. And the processes present in the root Cgroup makes the processes to have access to all CPUs in the system including the isolated cores. Since the kernel processes on the root Cgroup cannot be controlled from user-space, a possible solution is to control the process CPU affinities from kernel space. In proposed embodiments, CPU Isolation Enhancer 204 may message the counterpart Kernel threads CPU controller 206 (kernelspace) component about the CPUs to be isolated, the Cgroup path of the critical application, and so on (possibly via Netlink sockets). The Kernel threads CPU controller 206 may have access to the “task_struct”, a data structure maintained by the Linux kernel on per-task basis. Each task_struct maintains a variable “cpus_ptr”, which holds the list of CPUs on which a task is allowed to be scheduled. In Linux terminology, this is a “bit mask” for CPU affinity of a process.

[0049] FIG. 8 illustrates a flowchart describing a sequence of action taken by Kernel threads CPU controller 206. The Kernel threads CPU controller 206 may manage the CPU affinity of each process in the root Cgroup. This may begin, for example, based on a netlink message 802 received from the CPU Isolation Enhancer 204, which may contain information about the application, such as PID, Cgroup path, and a list of Isolcpus allocated to the application. For each process in the root Cgroup (the loop may begin at 804):• The Kernel threads CPU controller 206 may check if the task’s CPU affinity mask contains the isolated CPUs assigned to critical application, at 806.• If the CPU affinity mask of the process contains isolated CPUs, then Kernel threads CPU controller 206 may modify the “cpus_ptr” of that process to exclude the isolated CPUs, at 808.• If the CPU affinity mask of the process does not contain isolated CPUs, then Kernel threads CPU controller 206 may do nothing. At 810, if the list is not at its end, the next process is processed at 806, otherwise the success or error status may be returned.

[0050] By this method, processes in the root Cgroup can be controlled from getting scheduled on isolated CPUs. Note: There are per-core kernel worker threads performing the housekeeping job for efficient functioning of the Linux system. An instance of the kernel worker thread runs on every CPU in the system. For example, migration / x, ksoftirqd / x, rcuop / x,cpuhp / x, idle_inject / x, kworker / x, and so on, where “x” denotes the CPU number, are some of the kernel worker threads. When modifying the task affinities, Kernel threads CPU controller 206 does not change the CPU affinity of the per-core worker threads.

[0051] 3 -4. Periodic scan of all process in the system. FIG. 9 illustrates a flowchart describing a sequence of action taken by Kernel threads CPU controller 206 to do a periodic scan of all processes in the system to verify if any non-critical application has access to isolated CPUs allocated for critical application.

[0052] The sequence of actions described above are adequate to enhance the isolation when provisioning the critical application over isolated CPUs. But it is recommended to periodically check if any process in the system has gained access to the isolated CPUs. This can be done by periodically scanning the all the processes in the system. CPU Isolation Enhancer 204 may periodically instruct the Kernel threads CPU Controller 206 to conduct a periodic scan. The interval for periodic scan can be configured by a system administrator.

[0053] The Kernel threads CPU controller 206 may manage the CPU affinity of each process in the system. This may begin, for example, based on a netlink message 902 received from the CPU Isolation Enhancer 204, which may contain information about the application, such as PID, Cgroup path, and a list of Isolcpus allocated to the application. For each process in the system (the loop may begin at 904):• The Kernel threads CPU controller 206 may fetch the PID’s Cgroup, at 906.• The Kernel threads CPU controller 206 may check if Cgroup is associated with, or belongs to any critical application, at 908.• If the Cgroup is not associated with, or does not belong to any critical application, then Kernel threads CPU controller 206 may modify the “cpus_ptr” of that process to exclude the isolated CPUs, at 910.If the Cgroup is associated with, or does belong to any critical application, then Kernel threads CPU controller 206 may do nothing. At 912, if the list is not at its end, the next process is processed at 906, otherwise the success or error status may be returned.

[0054] Results from an experiment using an embodiment of the CPU Isolation Enhancer 204 are provided. The following command prompt listing shows a test application (critical application) running on isolated CPUs 30 and 134 as a container. It also shows corresponding Cgroup path for the test application. root:~ / projects / cpuirq-mt# kubectl get podsNAME READY STATUS RESTARTS AGE k8s-cpuirq-mt-b4f785d7f-tqd7n 1 / 1 Running 0 102m root:~ / projects / cpuirq-mt# root:~ / projects / cpuirq-mt# grep -r . / sys / fs / cgroup / kubepods.slice / kubepods- poddl066725_3fb3_4578_9a90_8c24fbd4c50e.slice / cri-containerd- 10bc3bl20d9fb4003e6f230e2e51f5dc3al31f7b0fblb3a7edf6b0d22b4ef29a.scope / cpuset.cpus30,134 root:~ / projects / cpuirq-mt#

[0055] The following command prompt listing shows the affinity error thrown by the task scheduler when another application tries to run on isolated CPUs 30 and 134 where the critical application is actively running on isolated cores. This ensures that the proposed component Cgroups Controller 208 can efficiently shield other processes from getting scheduled on isolated cores. This addresses the challenges mentioned above. root:~ / projects / cpuirq-mt# taskset -c 30 . / test-app-instance-2 -c 30 taskset: failed to set pid 639249’s affinity: Invalid argument root:~ / projects / cpuirq-mt# taskset -c 134 . / test-app-instance-2 -c 134 taskset: failed to set pid 639358’s affinity: Invalid argument root:~ / projects / cpuirq-mt#

[0056] The following command prompt listing shows a snip from a command that outputs all the process in the system and their corresponding CPUs allowed list. In comparison tothe similar command prompt listing above, where certain kernel processes (PIDs: 1,3,4,5,6,11,12,13,14,16,17,18 and 28) which had access to isolated cores before using the Kernel threads CPU controller 206, is now shielded from isolated CPU cores 30 and 134 when using Kernel threads CPU controller 206. This addresses the challenges discussed above. root:~ / projects / cpuirq-mt# for PID in Ts / proc / I grep -E ' [0-9]+$’ I sort -nkl,l'; do cpus='grep -r . / proc / $PID / status I grep Cpus_allowed_lisf ; echo PID-"$PID: $cpus”; donePID-1: Cpus_allowed_list: 0-299,31-133,135-207PID-2: Cpus_allowed_list: 0-29,31-133,135-207PID-3: Cpus_allowed_list: 0-29,31-133,135-207PID-4: Cpus_allowed_list: 0-29,31-133,135-207PID-5: Cpus_allowed_list: 0-29,31-133,135-207PID-6: Cpus_allowed_list: 0-29,31-133,135-207PID-8: Cpus_allowed_list: 0PID-10: Cpus_allowed_list: 0-29,133,135-207PID- 11 : Cpus_allowed_list: 0-29,31-133,135-207PID-12: Cpus_allowed_list: 0-29,31-133,135-207PID- 13: Cpus_allowed_list: 0-29,31-133,135-207PID- 14: Cpus_allowed_list: 0-29,31-133,135-207PID- 15: Cpus_allowed_list: 0PID- 16: Cpus_allowed_list: 0-29,31-133,135-207PID- 17: Cpus_allowed_list: 0-29,31-133,135-207PID- 18: Cpus_allowed_list: 0-29,31-133,135-207PID- 19: Cpus_allowed_list: 0PID-20: Cpus_allowed_list: 0PID-21: Cpus_allowed_list: 0PID-22: Cpus_allowed_list: 1PID-23 : Cpus_allowed_list: 1PID-24: Cpus_allowed_list: 1PID-25 : Cpus_allowed_list: 1PID-27 : Cpus_allowed_list: 1PID-28: Cpus_allowed_list: 0-29,31-133,135-207PID-29: Cpus_allowed_list: 2PID-30: Cpus_allowed_list: 2PID-31 : Cpus_allowed_list: 2PID-32: Cpus_allowed_list: 2PID-34: Cpus_allowed_list: 2

[0057] When the critical application runs to completion or is killed, the CPU Isolation Enhancer 204 may revert the steps 3-1, 3-2, and 3-3 identified above, returning the CPU to their unenhanced state. For example, for reverting steps 3-1 and 3-2, Cgroups Controller 208 may reassign the Isolcpus (where the critical application was provisioned) to other Cgroups folders which had the access to Isolcpus earlier. This may be done by appending the Isolcpus to cpuset. cpus file in the corresponding Cgroup folders. For reverting step 3-3, Kernel threads CPU controller 206 may modify the cpus_ptr variable corresponds to each kernel process in the root Cgroup to include the Isolcpus.

[0058] Variations to the embodiments disclosed above are possible. For example, embodiments can work on existing workloads (that is, workloads that are already scheduled on isolated cores) or for the applications that are about to get scheduled on the isolated cores. In certain scenarios, CPU Isolation Enhancer 204 can be extended to manage and allocate Isolcpus for the applications. Embodiments can work on both boot-time Isolcpus and dynamically createdIsolcpus at runtime. In cloud environments, CPU Isolation Enhancer 204 may be implemented as a system-daemon running in a node, or it may be implemented as part of Kubelet in Kubernetes, or it may be implemented as plugins in container runtimes like Containerd or CRI-O, among other possibilities.

[0059] Example

[0060] The following are results from an example conducted with an embodiment. In the test system, at boot-time, CPUs 30 and its hyper-thread sibling 134 are booted as isolated CPUs. The following command listing shows the grub parameters used at boot-time to create the Isolcpus (shown in bold). root:-# cat / proc / cmdlineBOOT_IMAGE= / boot / vmlinuz-5.19.0-45-generic root-UUID=fe459b3d-88ee- 449c-90e5-6fbd9116c85f ro audit=0 tsc=reliable skew_tick=l intel_iommu=on iommu=pt pcie_aspm=off default_hugepagesz=lG hugepagesz=lG hugepages=64 isolcpus=30,134 nohz_full=30,134 rcu_nocbs=30,134 rcu_nocb_poll nosoftlockup mceignore_ce lapic acpi_irq_nobalance nmi_watchdog=0 rdt=cmt,mbmtotal,mbmlocal,13cat,13cdp,12cat,12cdp,mba system.unified_cgroup_hierarchy=l nopti mitigations=off spectre_v2=off spec_store_bypass_disable=on lift=off megaraid_sas.msix_vectors= 1 vt.handoff=7

[0061] Two identical CPU intensive applications “test-app-instance-1” and “test-app- instance-2” were used for the demonstration. These applications periodically report OS jitter, which is a measure of time corresponding to the CPU cycles missed by a critical application running on a specific CPU core, as the missing CPU cycles would have been instead used by other co-hosted applications or kernel processes.

[0062] The following command prompt listings show a very low CPU jitter (close to -Ops) experienced by “test-app-instance-1” running on isolated CPUs 30 and 134. root:~ / projects / cpuirq-mt# . / test-app-instance-1 -c 30,134Specified CPU list: 30, 134, length: 7200 secondsStarting thread on core 30Starting thread on core 134Threads created, let the cores spin for 2 seconds before starting measurements.Starting jitter measurements.( 0) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 0;(#loops: 119754074)( 0) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 0;(#loops: 119754072)( 1) cpu: 134; Frequency: 1200153 kHz; max us (Is); 8; (from start); 8;(#loops: 119264667)( 1) cpu: 30; Frequency: 1200153 kHz; max us (Is); 12; (from start); 12;(#loops: 119263774)( 2) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8;(#loops: 119376428)( 2) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12;(#loops: 119376466)( 3) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8;(#loops: 119667638)( 3) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12;(#loops: 119667649)( 4) cpu: 134; Frequency: 1200153 kHz; max us (Is); 1; (from start); 8;(#loops: 116819396)( 4) cpu: 30; Frequency: 1200153 kHz; max us (Is); 1; (from start); 12;(#loops: 116819381)( 5) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119175375)( 5) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119175392)( 6) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119036121)( 6) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119036125)( 7) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119201936)( 7) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119201942)( 8) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119753814)( 8) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119753867)( 9) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119554975)( 9) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119554962)( 10) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 118763710)( 10) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 118763693)( 11) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 118871340)( 11) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 118871359)( 12) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119755077)( 12) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119755067)

[0063] FIG. 10 illustrates the low CPU jitter experienced by test-app-instance-1 on CPUs 30 and 134.

[0064] The following command prompt listing shows the 2nd application “test-app- instance-2” is scheduled and successfully runs on the same Isolcpus 30 and 134 where “test-app- instance-1” is running. root:~ / projects / cpuirq-mt# . / test-app-instance-2 -c 30,134Specified CPU list: 30, 134, length: 7200 secondsStarting thread on core 30Starting thread on core 134Threads created, let the cores spin for 2 seconds before starting measurements.Starting jitter measurements.( 0) cpu: 134; Frequency: 1200128 kHz; max us (Is); 16003; (from start); 16003; (#loops: 58946172)( 0) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 60665791)( 1) cpu: 134; Frequency: 1200128 kHz; max us (Is); 16002; (from start); 16003; (#loops: 60647490)( 1) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start);20002; (#loops: 61139838)( 2) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 60317478)( 2) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 60198639)( 3) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61678608)( 3) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20004; (from start); 20004; (#loops: 61614974)( 4) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61273981)( 4) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61321087)( 5) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61426380)( 5) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61498620)( 6) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61305504)( 6) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61320410)( 7) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61505040)( 7) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61624270)( 8) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61527503)( 8) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61528489)( 9) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20004; (from start); 20004; (#loops: 61333547)( 9) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61273123)( 10) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61482992)( 10) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20001; (from start); 20004; (#loops: 61505667)( 11) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20002; (from start); 20004; (#loops: 61398713)( 11) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20001; (from start); 20004; (#loops: 61477529)( 12) cpu: 134; Frequency: 1200128 kHz; max us (Is); 20001; (from start); 20004; (#loops: 59855661)( 12) cpu: 30; Frequency: 1200128 kHz; max us (Is); 20001; (from start); 20004; (#loops: 59910710)

[0065] The following command prompt listing shows the 1st application “test-app- instance-1” is scheduled and successfully runs on the same Isolcpus 30 and 134 where “test-app- instance-2” is running. root:~ / projects / cpuirq-mt# . / test-app-instance- 1 -c 30,134Specified CPU list: 30, 134, length: 7200 secondsStarting thread on core 30Starting thread on core 134Threads created, let the cores spin for 2 seconds before starting measurements.Starting jitter measurements.( 0) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 0;(#loops: 119754074)( 0) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 0;(#loops: 119754072)( 191) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119430300)( 191) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119430317)( 192) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119617114)( 192) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119617154)( 193) cpu: 134; Frequency: 1200153 kHz; max us (Is); 0; (from start); 8; (#loops: 119335269)( 193) cpu: 30; Frequency: 1200153 kHz; max us (Is); 0; (from start); 12; (#loops: 119335287)( 194) cpu: 134; Frequency: 1200153 kHz; max us (Is); 16002; (from start); 16002; (#loops: 99718399)( 194) cpu: 30; Frequency: 1200153 kHz; max us (Is); 16002; (from start);16002; (#loops: 99842553)( 195) cpu: 134; Frequency: 1200153 kHz; max us (Is); 16002; (from start); 16002; (#loops: 59990861)( 195) cpu: 30; Frequency: 1200153 kHz; max us (Is); 16002; (from start); 16002; (#loops: 59983854)( 196) cpu: 134; Frequency: 1200153 kHz; max us (Is); 16006; (from start); 16006; (#loops: 61081510)( 196) cpu: 30; Frequency: 1200153 kHz; max us (Is); 16004; (from start); 16004; (#loops: 61215919)( 197) cpu: 134; Frequency: 1200153 kHz; max us (Is); 16003; (from start); 16006; (#loops: 59900921)( 197) cpu: 30; Frequency: 1200153 kHz; max us (Is); 20001; (from start); 20001; (#loops: 61525849)( 198) cpu: 134; Frequency: 1200153 kHz; max us (Is); 20002; (from start); 20002; (#loops: 60669859)( 198) cpu: 30; Frequency: 1200153 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61047556)( 199) cpu: 134; Frequency: 1200153 kHz; max us (Is); 20002; (from start); 20002; (#loops: 60157865)( 199) cpu: 30; Frequency: 1200153 kHz; max us (Is); 20001; (from start); 20002; (#loops: 60399407)( 200) cpu: 134; Frequency: 1200153 kHz; max us (Is); 20002; (from start); 20002; (#loops: 61496260)( 200) cpu: 30; Frequency: 1200153 kHz; max us (Is); 20001; (from start); 20002; (#loops: 61575938)

[0066] FIG. 11 illustrates a spike in the CPU jitter experienced by “test-app-instance-1” when “test-app-instance-2” is co-located on the CPUs.

[0067] FIG. 12 is a flowchart illustrating a process 1200 for providing isolation from an operating system (OS) scheduler, according to an embodiment. Process 1200 may begin in step sl202, and may be performed, e.g., by a server 202, CPU Isolation Enhancer 204, kernel threads CPU controller 206, and / or Cgroups controller 208.

[0068] Step si 202 comprises determining information about critical processes associated with an application to be assigned to isolated CPUs.

[0069] Step sl204 comprises determining a set of CPUs to be isolated.

[0070] Step sl206 comprises, after boot-time, performing one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

[0071] In some embodiments, determining information about critical processes associated with an application to be assigned to isolated CPUs comprises determining one or more process ids (PIDs) and / or a Cgroup path associated with the application. In some embodiments, performing one or more isolation enhancements comprises (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated. In some embodiments, performing one or more isolation enhancements comprises (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated. In some embodiments, performing one or more isolation enhancements comprises both (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the process from running on the set of CPUs to be isolated.

[0072] In some embodiments, (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running onthe set of CPUs to be isolated comprises: (i) checking if a CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated; and (ii) if the CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated. In some embodiments, (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated comprises, for one or more non-migratable process of the one or more non-migratable processes (i) checking if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated; and (ii) if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated. In some embodiments, the method further includes periodically monitoring running processes and, for each process in the running processes: (i) determining a control group associated with the process; (ii) if the control group is a non-root control group: (ii- 1) determining if the control group is associated with the application to be assigned to isolated CPUs; (ii-2) if the control group is not associated with the application to be assigned to isolated CPUs, checking if a CPU affinity mask for the control group contains any CPU in the set of CPUs to be isolated and, if so, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated; (iii) if the control group is a root control group: (iii-1) if the process is a migratable process, migrating the process from the control group to a non-root control group to prevent the migratable process from running on the set of CPUs to be isolated; and (iii-2) if the process is a non-migratable process, checking if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated and if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated. In some embodiments, the method further includes, when the application is terminated or killed, reverting any changes caused by the one or more isolation enhancements.

[0073] FIG. 13 is a block diagram of apparatus 1300 (e.g., server 202, CPU Isolation Enhancer 204, kernel threads CPU controller 206, Cgroups controller 208), according to some embodiments, for performing the methods disclosed herein. As shown in FIG. 13, apparatus 1300 may comprise: processing circuitry (PC) 1302, which may include one or more processors(P) 1355 (e.g., a general purpose microprocessor and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., apparatus 1300 may be a distributed computing apparatus); at least one network interface 1348 comprising a transmitter (Tx) 1345 and a receiver (Rx) 1347 for enabling apparatus 1300 to transmit data to and receive data from other nodes connected to a network 1310 (e.g., an Internet Protocol (IP) network) to which network interface 1348 is connected (directly or indirectly) (e.g., network interface 1348 may be wirelessly connected to the network 1310, in which case network interface 1348 is connected to an antenna arrangement); and a storage unit (a.k.a., “data storage system”) 1308, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. Interface 1360 may connect PC 1302 and storage unit 1308, interface 1362 may connect PC 1302 and network interface 1348, and interface 1364 may connect network interface 1348 and network 1310. In embodiments where PC 1302 includes a programmable processor, a computer program product (CPP) 1341 may be provided. CPP 1341 includes a computer readable medium (CRM) 1342 storing a computer program (CP) 1343 comprising computer readable instructions (CRI) 1344. CRM 1342 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1344 of computer program 1343 is configured such that when executed by PC 1302, the CRI causes apparatus 1300 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1300 may be configured to perform steps described herein without the need for code. That is, for example, PC 1302 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0074] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described embodiments in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0075] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method for providing isolation from an operating system (OS) scheduler, the method comprising: determining information about critical processes associated with an application to be assigned to isolated CPUs; determining a set of CPUs to be isolated; after boot-time, performing one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

2. The method of claim 1 , wherein determining information about critical processes associated with an application to be assigned to isolated CPUs comprises determining one or more process ids (PIDs) and / or a Cgroup path associated with the application.

3. The method of any one of claims 1-2, wherein performing one or more isolation enhancements comprises (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated.

4. The method of any one of claims 1-3, wherein performing one or more isolation enhancements comprises (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable processes from running on the set of CPUs to be isolated.

5. The method of any one of claims 1-4, wherein performing one or more isolation enhancements comprises both (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable processes from running on the set of CPUs to be isolated.

6. The method of any one of claims 1-5, wherein (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated comprises:(i) checking if a CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated; and(ii) if the CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

7. The method of any one of claims 1-6, wherein (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated comprises, for one or more non-migratable process of the one or more non-migratable processeses:(i) checking if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated; and(ii) if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

8. The method of any one of claims 1-7, further comprising periodically monitoring running processes and, for each process in the running processes:(i) determining a control group associated with the process;(ii) if the control group is a non-root control group:(ii-1) determining if the control group is associated with the application to be assigned to isolated CPUs;(ii-2) if the control group is not associated with the application to be assigned to isolated CPUs, checking if a CPU affinity mask for the control group contains any CPU in the set of CPUs to be isolated and, if so, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated;(iii) if the control group is a root control group:(iii-1) if the process is a migratable process, migrating the process from the control group to a non-root control group to prevent the migratable process from running on the set of CPUs to be isolated; and(iii-2) if the process is a non-migratable process, checking if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated and if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

9. The method of any one of claims 1-8, further comprising, when the application is terminated or killed, reverting any changes caused by the one or more isolation enhancements.

10. A server (302) comprising:processing circuitry (1402); and a memory, the memory containing instructions (1444) executable by the processing circuitry (1402), whereby when executed the processing circuitry (1402) is configured to: determine information about critical processes associated with an application to be assigned to isolated CPUs; determine a set of CPUs to be isolated; after boot-time, perform one or more isolation enhancements, the isolation enhancements comprising: (i) migrating one or more migratable processes from a root control group to a nonroot control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

11. The server of claim 10, wherein determining information about critical processes associated with an application to be assigned to isolated CPUs comprises determining one or more process ids (PIDs) and / or a Cgroup path associated with the application.

12. The server of any one of claims 10-11, wherein performing one or more isolation enhancements comprises (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated.

13. The server of any one of claims 10-12, wherein performing one or more isolation enhancements comprises (ii) controlling one or more non-migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated.

14. The server of any one of claims 10-13, wherein performing one or more isolation enhancements comprises both (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated; and (ii) controlling one or more non-migratable processes running in the root control group to prevent the process from running on the set of CPUs to be isolated.

15. The server of any one of claims 10-14, wherein (i) migrating one or more migratable processes from a root control group to a non-root control group to prevent the migratable processes from running on the set of CPUs to be isolated comprises:(i) checking if a CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated; and(ii) if the CPU affinity mask for the non-root control group contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

16. The server of any one of claims 10-15, wherein (ii) controlling one or more non- migratable processes running in the root control group to prevent the non-migratable process from running on the set of CPUs to be isolated comprises, for one or more non-migratable process of the one or more non-migratable processes:(i) checking if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated; and(ii) if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modifying the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

17. The server of any one of claims 10-16, wherein the processing circuitry is further configured to periodically monitor running processes and, for each process in the running processes:(i) determine a control group associated with the process;(ii) if the control group is a non-root control group:(ii-1) determine if the control group is associated with the application to be assigned to isolated CPUs;(ii-2) if the control group is not associated with the application to be assigned to isolated CPUs, check if a CPU affinity mask for the control group contains any CPU in the set of CPUs to be isolated and, if so, modify the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated;(iii) if the control group is a root control group:(iii-1) if the process is a migratable process, migrate the process from the control group to a non-root control group to prevent the migratable process from running on the set of CPUs to be isolated; and(iii-2) if the process is a non-migratable process, check if a CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated and if the CPU affinity mask for the non-migratable process contains any CPU in the set of CPUs to be isolated, modify the CPU affinity mask to exclude any CPU in the set of CPUs to be isolated.

18. The server of any one of claims 10-17, wherein the processing circuitry is further configured to, when the application is terminated or killed, revert any changes caused by the one or more isolation enhancements.

19. A computer program (1443) comprising instructions which when executed by processing circuitry (1402) of a node (1400), causes the node (1400) to perform the method of any one of claims 1-9.

20. A carrier containing the computer program (1443) of claim 19, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1442).

Images