A method for optimizing percpu memory usage in Kylin V10 system

Abstract

The present application relates to a method for optimizing percpu memory usage in a Kirin V10 system, and to the field of computer technology, including when an upper-level user deletes a container, the system kernel deletes the group ID of the Cgroup group corresponding to the container, and encapsulates the operation of releasing the percpu memory into work, and adds it to the work queue. When an idle cpu is detected in the system, the cpu executes the work task to synchronize the percpu memory in the Cgroup group to the corresponding target upper-level Cgroup group, and releases various percpu memories in the Cgroup group. After the percpu memory is released, it is determined whether the process in the Cgroup group has file IO reading and writing. If so, it is determined whether the available memory in the current system is lower than the memory water level set by the system. If so, a memory recovery mechanism is triggered so that the current Cgroup group has been completely cleaned up.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for optimizing percpu memory usage in a Kirin V10 system. Background Art

[0002] Container technology provides an efficient method for deploying applications and realizes the isolation of resources in the system. It has been widely used in cloud computing, cluster management, microservices, databases and other fields. Container technology relies on the Cgroup group in the system to complete resource control. Each upper-layer container corresponds to a Cgroup group in the system. In some application scenarios where a large number of containers are created and deleted frequently, the percpu memory usage in the system often continues to increase, which leads to a gradual decrease in the available memory in the system. This problem is particularly prominent on devices with a large number of CPUs.

[0003] Percpu memory refers to the memory area allocated for each processor, which is usually used by the kernel to store its dedicated data structures and information for each CPU core. This part of memory is mainly used to optimize the performance of multi-core processors by reducing data sharing and competition between different CPU cores.

[0004] The main reason for this problem is that the cache pages managed by the Cgroup group are retained for a long time in the system. When the process in the container operates the file, the system will allocate cache pages to temporarily save the contents of the file in the storage medium, and put the cache pages into the Cgroup group corresponding to the container for management. In order to ensure IO read and write efficiency, the system will not release the cache pages immediately when the process manages the file, and the Cgroup group to which the cache pages belong will not be cleared. Even if the upper-level user deletes the container, the Cgroup group in the system will not be deleted, and it will become a zombie Cgroup group, which will be retained until the cache pages in the system are cleared.

[0005] When the device runs for a long time, the cache pages in the system will continue to increase. After a large number of containers are deleted and created, the number of zombie Cgroups in the system will gradually increase. Since the Cgroup contains a lot of percpu memory to store statistics to distinguish CPUs, the increasing number of zombie Cgroups will eventually lead to an increase in the percpu memory usage in the system.

[0006] This phenomenon occurs in the Kylin V10 system. For the sake of system IO read and write efficiency and container deletion and creation efficiency, the Linux kernel community retains the current design. If a device that has been running for a long time has a large number of zombie Cgroups because the cache pages cannot be released immediately, you can manually use / proc / sys / vm / drop_caches to release the cache.

[0007] For customers who have large-scale business deployment environments, in order to prevent this problem from affecting normal business in various scenarios, a cache page cleaning script must also be deployed during large-scale deployment. The script regularly cleans up cache pages on the device, deletes the zombie Cgroup group, and releases percpu memory. Manually forcibly cleaning up cache pages in the system will cause the system's IO read and write efficiency to decrease in the short term, affecting business capabilities in various application scenarios. At the same time, additional script operations will increase the complexity of large-scale deployment of upper-layer businesses. Summary of the invention

[0008] To solve the above problems, the present invention provides a method for optimizing the percpu memory usage in the Kirin V10 system, which adopts the following technical solutions:

[0009] A method for optimizing percpu memory usage in a Kirin V10 system, comprising:

[0010] When the upper-level user deletes the container, the system kernel deletes the group ID of the Cgroup group corresponding to the container, encapsulates the percpu memory release operation into work, adds the encapsulated work to the work queue, and sets the status value of the Cgroup group to 1, where the status value 1 indicates the disable state;

[0011] When it is detected that there is an idle CPU in the system that can process the encapsulated work, the CPU executes the work task to synchronize the statistical information stored in the percpu memory in the current Cgroup group to its corresponding target upper-level Cgroup group, and at the same time sets the status value of the Cgroup group to 2, where the status value 2 indicates the flushed state, and the statistical information includes: the number of anonymous pages, file pages, and dynamic pages used by the program in the Cgroup group;

[0012] After the Cgroup state switches to the flushed state, the work task immediately releases various percpu memories in the Cgroup. After all percpu memories are released, the Cgroup state value is set to 3, where the state value 3 indicates entering the freed state, indicating that the Cgroup is in a state of waiting to be completely released;

[0013] Determine whether the process in the current Cgroup group has file IO reading and writing;

[0014] If it does not exist, it is determined that the current Cgroup group has been completely cleaned up;

[0015] If it exists, it is determined whether the available memory in the current system is lower than the memory watermark set by the system. If it is lower, the memory recycling mechanism is triggered and the cache pages in the system are recycled so that the current Cgroup group has been completely cleaned up.

[0016] In another possible implementation, the encapsulated work is added to the work queue, including:

[0017] The encapsulated work is added to the work queue through the queue_work operation.

[0018] In another possible implementation, setting the status value of the Cgroup group to 1 includes:

[0019] Set the status value of the Cgroup group to 1 through disable_css.

[0020] In another possible implementation, the work task synchronizes the statistical information stored in the percpu memory in the current Cgroup group to its corresponding target upper-level Cgroup group, including:

[0021] By implementing flush_percpu_vmstats in the work task, the statistical information stored in the percpu memory in the current Cgroup group is synchronized to its corresponding target upper-level Cgroup group.

[0022] In another possible implementation, the statistics information stored in the percpu memory in the current Cgroup group is synchronized to the corresponding target upper-level Cgroup group by implementing flush_percpu_vmstats in the work task, including:

[0023] Step Sa, determining whether the status value of the upper layer Cgroup group of the current Cgroup group is 0, wherein the status value 0 indicates the enable state;

[0024] Step Sb: If yes, the upper layer Cgroup group is determined as the target upper layer Cgroup group corresponding to the current Cgroup group, and the statistical information stored in the percpu memory of the current Cgroup group is synchronized to the target upper layer Cgroup group;

[0025] Step Sc, if not, recursively search the upper layer Cgroup group of the upper layer Cgroup group until a Cgroup group with a status value of 0 is found, and determine it as the target upper layer Cgroup group of the current Cgroup group, and synchronize the statistical information stored in the percpu memory in the current Cgroup group to the target upper layer Cgroup group.

[0026] In another possible implementation, the work task immediately releases various percpu memory in the Cgroup group, including:

[0027] Free up various percpu memory in the Cgroup group by implementing the free_percpu operation in the work task.

[0028] In another possible implementation, determining whether a process in the current Cgroup group has file IO reading and writing includes:

[0029] By checking whether the number of references of the current Cgroup group is reduced to 0, it is determined whether the process in the current Cgroup group has file IO reading and writing. When the number of references is 0, it indicates that there is no file IO reading and writing. When the number of references is not 0, it indicates that there is file IO reading and writing.

[0030] In another possible implementation, the method further includes:

[0031] When it is determined that the available memory in the current system is not lower than the memory watermark set by the system, the upper-level user of the system can force the system to recycle cache pages through a memory recycling tool so that the Cgroup group is completely cleared.

[0032] In summary, the present application includes at least one of the following beneficial technical effects:

[0033] 1. In an environment where containers are frequently deleted in large quantities, the present invention can significantly and effectively alleviate the growth of percpu memory usage in the system and prevent excessive consumption of system memory.

[0034] 2. The core operation of the present invention is performed in an asynchronous processing manner, which reduces the efficiency of container deletion and also ensures the efficiency of file IO reading and writing in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a structural diagram of a method for optimizing percpu memory usage in a Kirin V10 system in an embodiment of the present application.

[0036] Figure 2It is a flowchart of a method for optimizing percpu memory usage in a Kirin V10 system in an embodiment of the present application.

[0037] Figure 3 It is a schematic diagram of the function call flow for synchronizing statistical information to the target upper-level Cgroup group in an embodiment of the present application.

[0038] Figure 4 It is a comparison chart of the effects of the present application and the original solution in the embodiments of the present application. DETAILED DESCRIPTION

[0039] The present application is further described in detail below in conjunction with the accompanying drawings.

[0040] After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed, but such modifications are protected by patent law as long as they are within the scope of the claims of this application.

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0043] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0044] For ease of understanding, the thumbnails and key terms involved in this application are first explained as follows:

[0045] Cgroup: Cgroup is the abbreviation of control groups, which is a mechanism in the Linux kernel used to limit, record, and isolate the physical resources (such as CPU, memory, IO, DMA, etc.) used by process groups. Cgroup is the foundation of container technology. Cloud native, cloud computing, big data hybrid deployment technology, etc. all use Cgroup as the technical foundation.

[0046] Cgroupfs: Cgroupfs is a file system for user-mode applications to interact with Cgroups in the kernel. A series of native interfaces provided by the kernel Cgroup are provided to applications through Cgroupfs.

[0047] Container: Container technology effectively divides the resources of a single operating system into independent groups to better balance conflicting resource usage requirements between independent groups. The processes within the group use resources independently and do not affect each other. This technology is called container technology.

[0048] Docker: Docker is an open source application container engine that allows developers to package their applications and dependent packages into a portable image and then publish them to any popular Linux or Windows operating system machine, and can also achieve virtualization. Containers use a complete sandbox mechanism and there is no interface between them.

[0049] K8s: kubernetes, or K8s for short, is an abbreviation that uses 8 to replace the eight characters in the middle of the name "ubernete". It is an open source containerized application management platform for multiple hosts. The goal of Kubernetes is to make the deployment of containerized applications simple and powerful. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance.

[0050] Kata: Kata Containers is an open source community working to build secure container runtimes using lightweight virtual machines that feel and perform like containers but provide stronger workload isolation using hardware virtualization technology as a second layer of defense.

[0051] Secondly, if Figure 1 As shown, various container components used in the business environment (such as docker, k8s, kata, etc.) will operate the cgroupfs file system in the operating system when deleting or creating containers, and then delete or create the Cgroup in the kernel. The embodiment of the present application provides a method for optimizing the percpu memory usage in the Kylin V10 system. Under the premise of ensuring that the overall deletion and creation process of the Cgroup group remains unchanged, two operations, disable_css and queue_work, are added to the kernel process. The main function of the disable_css operation is to set the status of the Cgroup group to disable, which is used to prohibit further modification operations on the current Cgroup group. The main function of the queue_work operation is to create a work and add this work to the work queue.

[0052] The kernel will schedule the work in the work queue and allocate idle CPUs to process it. The work has two main tasks: one is to synchronize the statistics stored in the percpu memory of the current Cgroup group to its corresponding target upper-level Cgroup, and the other is to release the percpu memory in the Cgroup group.

[0053] Since the disable_css and queue_work operations take very short time, their impact on the efficiency of the overall operation process of deleting Cgroup groups in the system can be almost ignored.

[0054] Specifically, Figure 2 As shown, in order to distinguish the operations to be performed by the Cgroup group at different times when deleting or creating containers, the present application adds a status flag to the Cgroup group. The state switching process in the embodiment of the present application is also an implementation process of a method for optimizing percpu memory usage in a Kylin V10 system, which is specifically as follows:

[0055] Step S1, when the upper-level user creates a container, the kernel uses cgroupfs to create a corresponding Cgroup group for the container. At this time, the status value of the Cgroup defaults to 0, indicating the enable state.

[0056] Step S2: When the upper-level user deletes the container, the system kernel deletes the group ID of the Cgroup group corresponding to the container (corresponding to Figure 2 1. Release cgroup id in ), encapsulate the percpu memory release operation into work, and add the encapsulated work to the work queue through the queue_work operation (corresponding to Figure 2 2. Add the work of releasing memory to the queue) and set the status value of the Cgroup group to 1 through disable_css, where the status value 1 indicates the disable state.

[0057] Step S3: When it is detected that there is an idle CPU in the system that can process the encapsulated work, the CPU executes the work task to synchronize the statistical information stored in the percpu memory of the current Cgroup group to its corresponding target upper-level Cgroup group (corresponding to Figure 2 The system schedules work in the queue in the cgroup, synchronizes the statistics in the cgroup to the parent cgroup), and sets the status value of the cgroup group to 2 at the same time. The status value 2 indicates the flushed state. The statistics include: the number of anonymous pages, file pages, dynamic pages and other memory-related information used by programs in the cgroup group.

[0058] For the embodiment of the present application, by implementing flush_percpu_vmstats in the work task, the statistical information stored in the percpu memory in the current Cgroup group is synchronized to its corresponding target upper-level Cgroup group, and the specific steps are as follows:

[0059] Step Sa, determining whether the status value of the upper layer Cgroup group of the current Cgroup group is 0, wherein the status value 0 indicates the enable state;

[0060] Step Sb: If yes, the upper layer Cgroup group is determined as the target upper layer Cgroup group of the current Cgroup group, and the statistical information stored in the percpu memory of the current Cgroup group is synchronized to the target upper layer Cgroup group;

[0061] Step Sc, if not, recursively search the upper layer Cgroup group of the upper layer Cgroup group until the Cgroup group with a status value of 0 is found, and determine it as the target upper layer Cgroup group of the current Cgroup group, and synchronize the statistical information stored in the percpu memory in the current Cgroup group to the target upper layer Cgroup group.

[0062] Furthermore, in an embodiment of the present application, the operation of synchronizing the statistical information stored in the percpu memory in the current Cgroup group to its target upper-level Cgroup group can be performed not only in the disable state of the Cgroup group, but also when the statistical information stored in the percpu memory in the Cgroup group is updated, as long as the state value of the Cgroup group is not 0 (that is, in any of the states of disable, flushed, and freed), steps Sa to Sc will be executed to accumulate the updated statistical information into its corresponding target upper-level Cgroup group. The specific function call process is as follows: Figure 3 shown.

[0063] Step S4, after the Cgroup state is switched to the flushed state, various percpu memories in the Cgroup group are released by implementing the free_percpu operation in the work task (corresponding to Figure 2 After all percpu memories are released, the Cgroup status value is set to 3, where a status value of 3 indicates entering the freed state, indicating that the Cgroup is waiting to be completely released.

[0064] Through steps S1 to S4, the present application can greatly reduce the occupation of percpu memory resources in the zombie Cgroup group, so that the occupation of system memory resources by the zombie Cgroup group during the zombie period is minimized. However, since there may be cache pages in the Cgroup group at this time to improve the IO read and write efficiency of the system, it is necessary to further determine whether the current Cgroup group has been completely cleaned up, specifically:

[0065] Through step S5, it is determined whether there is file IO reading and writing in the process of the current Cgroup group. When there is no file IO reading and writing, it means that there is no corresponding cache page in the current Cgroup group. Therefore, it can be determined that after the percpu occupied in the kernel layer Cgroup group corresponding to the container is released through step S4, the Cgroup group itself is also directly and completely cleaned up; and when there is still file IO reading and writing, it means that in order to improve IO efficiency, the system kernel allows the cache page of the Cgroup group to reside in the system, that is, after the percpu occupied in the kernel layer Cgroup group corresponding to the container is released through step S4, the Cgroup group itself is not completely cleaned up. Therefore, at this time, it can be further determined whether the available memory in the current system is lower than the memory waterline set by the system. When it is lower, the memory recovery mechanism is triggered, and the corresponding Cgroup group is completely cleaned up after the cache page in the system is recovered. Further, when it is not lower than, the upper-level users of the system can also use the memory recovery tool to force the system to recycle the cache page, and the Cgroup group will also be completely cleaned up.

[0066] Among them, since the reference count of the corresponding kernel layer Cgroup group will decrease when the user layer container is deleted, it is possible to determine whether the process in the current Cgroup group has file IO reading and writing by checking whether the reference count of the current Cgroup group is reduced to 0. When the reference count is 0, it means that there is no file IO reading and writing, and when the reference count is not 0, it means that there is file IO reading and writing.

[0067] In an environment where containers are frequently deleted in large quantities, this application can significantly and effectively alleviate the growth of percpu memory usage in the system and prevent excessive consumption of system memory. And because the core operation is performed in an asynchronous processing manner, the impact on the efficiency of container deletion is reduced, while also ensuring the efficiency of file IO reading and writing in the system. Specifically, the percpu memory growth rate after using the optimization method in this application and the percpu memory growth rate of the original solution are compared as shown in the figure below. Figure 4 As shown,

[0068] Line 1 shows the growth trend of the number of all Cgroups in the system, including zombie Cgroups.

[0069] Line 2 shows the percpu memory increase trend chart in the system after using the present invention.

[0070] Line 3 shows the percpu memory increase trend in the system before using the present invention.

[0071] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0072] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for optimizing percpu memory usage in a Kylin V10 system, characterized in that: include: When the upper-level user deletes the container, the system kernel deletes the group ID of the Cgroup group corresponding to the container, encapsulates the percpu memory release operation into work, adds the encapsulated work to the work queue, and sets the status value of the Cgroup group to 1, where the status value 1 indicates the disable state; When it is detected that there is an idle CPU in the system that can process the encapsulated work, the CPU executes the work task to synchronize the statistical information stored in the percpu memory in the current Cgroup group to its corresponding target upper-level Cgroup group, and at the same time sets the status value of the Cgroup group to 2, where the status value 2 indicates the flushed state, and the statistical information includes: the number of anonymous pages, file pages, and dynamic pages used by the program in the Cgroup group; After the Cgroup state switches to the flushed state, the work task immediately releases various percpu memories in the Cgroup. After all percpu memories are released, the Cgroup state value is set to 3, where the state value 3 indicates entering the freed state, indicating that the Cgroup is in a state of waiting to be completely released; Determine whether the process in the current Cgroup group has file IO reading and writing; If it does not exist, it is determined that the current Cgroup group has been completely cleaned up; If it exists, it is determined whether the available memory in the current system is lower than the memory watermark set by the system. If it is lower, the memory recycling mechanism is triggered and the cache pages in the system are recycled so that the current Cgroup group has been completely cleaned up.

2. According to claim 1, a method for optimizing percpu memory usage in a Kylin V10 system, characterized in that: Add the encapsulated work to the work queue, including: The encapsulated work is added to the work queue through the queue_work operation.

3. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 1, characterized in that: Set the status value of the Cgroup group to 1, including: Set the status value of the Cgroup group to 1 through disable_css.

4. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 1, characterized in that: The work task synchronizes the statistics information stored in the percpu memory in the current Cgroup group to its corresponding target upper-level Cgroup group, including: By implementing flush_percpu_vmstats in the work task, the statistical information stored in the percpu memory in the current Cgroup group is synchronized to its corresponding target upper-level Cgroup group.

5. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 4, characterized in that: By implementing flush_percpu_vmstats in the work task, the statistics stored in the percpu memory in the current Cgroup group are synchronized to its corresponding upper-level Cgroup group, including: Step Sa, determining whether the status value of the upper layer Cgroup group of the current Cgroup group is 0, wherein the status value 0 indicates the enable state; Step Sb: If yes, the upper layer Cgroup group is determined as the target upper layer Cgroup group corresponding to the current Cgroup group, and the statistical information stored in the percpu memory in the current Cgroup group is synchronized to the target upper layer Cgroup group; Step Sc, if not, recursively search the upper layer Cgroup group of the upper layer Cgroup group until the Cgroup group with a status value of 0 is found, and determine it as the target upper layer Cgroup group of the current Cgroup group, and synchronize the statistical information stored in the percpu memory in the current Cgroup group to the target upper layer Cgroup group.

6. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 1, characterized in that: The work task immediately releases various percpu memory in the Cgroup group, including: Free up various percpu memory in the Cgroup group by implementing the free_percpu operation in the work task.

7. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 1, characterized in that: Determine whether the process in the current Cgroup group has file IO reading and writing, including: By checking whether the number of references of the current Cgroup group is reduced to 0, it is determined whether the process in the current Cgroup group has file IO reading and writing. When the number of references is 0, it indicates that there is no file IO reading and writing. When the number of references is not 0, it indicates that there is file IO reading and writing.

8. The method for optimizing percpu memory usage in a Kylin V10 system according to claim 1, characterized in that: The method further comprises: When it is determined that the available memory in the current system is not lower than the memory watermark set by the system, the upper-level user of the system can force the system to recycle cache pages through a memory recycling tool so that the Cgroup group is completely cleared.

Images