Memory overflow event processing method and device, electronic equipment and storage medium

Abstract

The invention relates to a memory overflow event processing method and device, electronic equipment and a storage medium, and relates to the field of memory overflow event processing. The method comprises the following steps: in a kernel mode, capturing a memory overflow event in real time through a detection function pre-mounted on a preset function; in response to the captured memory overflow event, collecting related data of the memory overflow event, and sending the related data to the user mode; in the user mode, performing enrichment processing on the related data of the memory overflow event to obtain a memory overflow event log; and reporting the memory overflow event log. By applying the method and the device, the information of the memory overflow event can be efficiently and accurately acquired in real time, and delay and missing report are avoided.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for processing a memory overflow event. Background Art

[0002] With the continuous development of cloud-native technology, more and more enterprises are migrating their businesses to containerized infrastructure. Cloud-native architecture manages containerized applications through container orchestration systems such as Kubernetes, greatly improving the flexibility, scalability and resource utilization efficiency of the system. The development of cloud-native technology has promoted the widespread application of container technology, making containers the standard for modern business deployment. However, in cloud-native scenarios, resource management and performance monitoring have become important guarantees for the normal operation of businesses, especially the management of memory resources. In a cloud-native environment, the allocation and use of memory resources are controlled by cgroups (control groups), which define resource limits for each container, including CPU, memory, network, I / O, etc. When the memory resources used by a container exceed its predetermined memory resources, the kernel triggers an OOM (OutOf Memory) event and forcibly kills certain memory-consuming processes to release memory resources. With the increase in the amount of container data, especially in a dynamically scalable cloud environment, OOM events become more frequent, which poses a challenge to system stability and business continuity. OOM events are particularly common in cloud-native environments. Due to the dynamic scheduling of containers on platforms such as Kubernetes, large fluctuations in business pressure, and unpredictable container memory consumption, the occurrence of OOM events has become more uncontrollable. OOM events can cause problems such as business interruption, difficulty in problem diagnosis, and complex business impact. Therefore, in a cloud-native environment, real-time monitoring of OOM events is particularly important. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, device, electronic device, and storage medium for processing a memory overflow event.

[0004] In a first aspect, an embodiment of the present disclosure provides a method for handling a memory overflow event, comprising: in a kernel state, capturing a memory overflow event in real time by means of a detection function pre-mounted in a preset function; in response to capturing a memory overflow event, collecting relevant data of the memory overflow event, and sending the relevant data to a user state; in the user state, enriching the relevant data of the memory overflow event to obtain a memory overflow event log; and reporting the memory overflow event log.

[0005] In the second aspect, an embodiment of the present disclosure provides a memory overflow event processing device, including: an event capture unit, configured to capture memory overflow events in real time in kernel state through a detection function pre-mounted in a preset function; a data collection unit, configured to collect relevant data of the memory overflow event in response to capturing the memory overflow event, and send the relevant data to the user state; a data enrichment unit, configured to enrich the relevant data of the memory overflow event in the user state to obtain a memory overflow event log; and a log reporting unit, configured to report the memory overflow event log.

[0006] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the memory overflow event handling method described in the first aspect when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the memory overflow event processing method as described in the first aspect.

[0008] By applying the technical solution disclosed in the present invention, relevant data on memory overflow time can be collected by mounting a detection function on a preset function in the kernel state, and the relevant data can be sent to the user state. The relevant data can be enriched in the user state to obtain a memory overflow event log, and the memory overflow event log can be further reported, so that information on memory overflow events can be obtained in real time, efficiently and accurately, avoiding delays and omissions.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 An exemplary system architecture diagram to which an embodiment of the memory overflow event processing method disclosed herein can be applied; Figure 2 A flowchart of an embodiment of a method for handling a memory overflow event disclosed herein; Figure 3 A flowchart of another embodiment of the method for handling a memory overflow event disclosed herein; Figure 4 A schematic diagram of the structure of an embodiment of a memory overflow event processing device disclosed in the present invention; Figure 5A schematic diagram of the structure of an embodiment of an electronic device disclosed herein. DETAILED DESCRIPTION

[0011] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0012] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0013] In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0014] In order to make the technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0015] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of a method or apparatus for handling a memory overflow event disclosed in the present invention can be applied.

[0016] like Figure 1 As shown, the system architecture 100 may include a node 101, a network 102, a server 103, and a database 104. The network 102 is used to provide a medium for a communication link between the node 101 and the server 103 and the database 104. The network 102 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0017] The operating system of node 101 may include user state and kernel state. Among them, the CPU in kernel state can access any data and peripheral devices, and can also switch from one program to another program, and occupying the CPU will not cause preemption. The CPU in user state can only access limited resources, and cannot directly access hardware devices such as memory. It must be trapped in the kernel through system calls to access these privileged resources. Node 101 can upload the log of the node overflow event to database 104 when a node overflow event occurs. Node 101 can be connected to a display device to display information such as the log of the node overflow time.

[0018] A node is a working node in a Kubernetes cluster that is responsible for running containerized applications. Each node has a runtime environment (such as Docker) and runs multiple Pods. A node can be a physical machine or a virtual machine. A Pod is the smallest schedulable unit in Kubernetes and usually contains one or more containers that share storage, network, and configuration. The containers in a Pod collaborate to complete a task and are always scheduled and run together.

[0019] The server 103 can obtain the log of the node overflow event from the database 104, analyze the log, and feed back the analysis result to the node 101. The node 101 can adjust the configuration information according to the result fed back by the server 103.

[0020] The database 104 may be any type of database, which may store logs of node overflow events. The database 103 may be accessed at a specified time or under specified conditions, and the node 101 may also access the database 104 .

[0021] It should be noted that part of the steps of the memory overflow event processing method provided in the embodiment of the present disclosure can be executed by the node 101, and another part of the steps can be executed by the server 103. Or all the steps of the memory overflow event processing method are executed by the node 101. Accordingly, part of the memory overflow event processing device can be set in the node 101, and another part can be set in the server 103, or the entire memory overflow event processing device is set in the node 101.

[0022] It should be understood that Figure 1 The number of nodes, networks, servers and databases in the embodiment is only for illustration. Any number of nodes, networks, servers and databases may be provided as required.

[0023] Figure 2 FIG. 200 shows a process 200 of an embodiment of a method for handling a memory overflow event of the present disclosure. Figure 2 As shown, the memory overflow event processing method of this embodiment may include the following steps: Step 201, in kernel state, a memory overflow event is captured in real time by a detection function pre-mounted in a preset function.

[0024] In this embodiment, the execution subject of the memory overflow event processing method (for example Figure 1The node 101 shown in FIG. 101 can capture memory overflow events in real time through a preset function. Specifically, the preset function can be a function for handling memory overflow events. In some specific practices, the preset function can be the oom_kill_process function, which is a key function used by the kernel to handle OOM events. A detection function can be pre-mounted in the above preset function. When an OOM event occurs, the detection function will sense it, thereby enabling real-time capture of the OOM event. Specifically, the detection function can be mounted in the form of a hook function.

[0025] Step 202, in response to capturing a memory overflow event, collecting data related to the memory overflow event, and sending the data related to the user state.

[0026] In this embodiment, the relevant data of the OOM event can be collected before or when the preset function processes the OOM event. The above-mentioned relevant data may include the context of the OOM event, and may also include the thread that triggered the OOM event, and the information of the process killed by the preset function. After the above-mentioned relevant data is obtained, the above-mentioned relevant data can be sent to the user state through the communication channel between the kernel state and the user state.

[0027] In some specific practices, eBPF can be used to insert probes into the above oom_kill_process function to collect relevant data. eBPF stands for extended Berkeley Packet Filter. The predecessor of eBP technology is called BPF (Berkeley Packet Filter) or cBPF (classic BPF), which was first mentioned in a paper by Steven McCanne and Van Jacobson in 1992, "The BSD Packet Filter: A New Architecture for User-level Packet Capture". In short, eBPF is a very flexible and efficient virtual machine-like component in the Linux kernel that can safely execute bytecode at many kernel hook points. Many kernel subsystems have already used BPF, such as common networking, tracing and security. Accordingly, the perf buffer mechanism of eBPF can be used to send the captured OOM event related data to the user state for processing. The perf buffer mechanism is an efficient mechanism for safely and efficiently transmitting the data collected by the eBPF program to the user space. It is based on the Perf Events subsystem in the Linux kernel and provides a way for eBPF programs to pass events or other data to processes running in user space.

[0028] Step 203: In the user state, enrich the data related to the memory overflow event to obtain a memory overflow event log.

[0029] In user mode, the relevant data of OOM events can be enriched to obtain memory overflow event logs. Data enrichment refers to the process of enriching original data by adding or supplementing additional information. This includes calling external services to obtain additional information, cleaning and verifying data, format conversion, etc., to improve the integrity and applicability of data. Specifically, according to the specified fields in the relevant data, the fields related to the above specified fields can be found, and the correlation between the above specified fields and the found fields can be established to obtain the OOM event log.

[0030] Step 204: Report the memory overflow event log.

[0031] After obtaining the memory overflow event log, it can be reported to the database, so that it can be read for further processing.

[0032] The memory overflow event processing method provided by the above-mentioned embodiment of the present disclosure can collect relevant data of memory overflow time by mounting a detection function on a preset function in the kernel state, and send the relevant data to the user state, and enrich the relevant data in the user state to obtain a memory overflow event log, and further report the memory overflow event log, so as to obtain information on memory overflow events in real time, efficiently and accurately, avoiding delays and omissions.

[0033] Continue to see Figure 3 , which shows a process 300 of another embodiment of the memory overflow event processing method according to the present disclosure. Figure 3 As shown, the memory overflow event processing method of this embodiment may include the following steps: Step 301, insert a probe at a preset function, and mount a detection function at the probe.

[0034] In this embodiment, on each node, an eBPF kprobe can be used in user mode to insert a probe into the kernel function oom_kill_process, which is specifically used to handle the process termination event of insufficient memory. This probe mounting allows real-time capture of OOM events occurring in the system.

[0035] Step 302: In kernel mode, a detection function pre-installed in a preset function is used to capture a memory overflow event in real time.

[0036] By pre-mounting the detection function on the kernel function oom_kill_process, OOM events can be detected in real time.

[0037] Step 303, in response to capturing the memory overflow event, collecting data related to the memory overflow event, and sending the related data to the user state.

[0038] After the OOM event is sensed, the OOM event context information can be obtained, including: the PID, NSPID, process path, process name, container ID of the process that triggered the OOM event, the number of memory pages occupied by the process, the parent process PID, parent process NSPID, parent process name, parent process path, and the PID, NSPID, process path, process name, container ID of the process, parent process PID, parent process NSPID, parent process name, and parent process path of the process killed by OOM. See Table 1 for details. The above collected information is sent to the user state through the perf buffer for further processing. This process runs in the kernel state through the eBPF program, which can achieve real-time OOM event capture with extremely low performance overhead.

[0039] Table 1. Data related to collected OOM events

[0040] Step 304, in the user state, determine the container identifier from the relevant data; determine the container metadata corresponding to the container identifier according to the container identifier; and enrich the relevant data according to the container metadata to obtain a memory overflow event log.

[0041] In this embodiment, the user state can cache the metadata of the container and the metadata of the process. In the Kubernetes environment, Pod is the smallest resource management component. Pod metadata refers to various information related to the Pod in the Kubernetes environment, including the name of the Pod, IP address, namespace, running node name, service account name, CPU and memory usage and limits of the container, Pod labels and annotations, etc. When receiving the relevant data of the OOM event, the data can be enriched through the container identifier (i.e., container_id) in the relevant data. In some specific practices, the enriched data includes the container name, Pod name and namespace, and the finally generated OOM event log is more complete. The data included in the enriched OOM event log is shown in Table 2.

[0042] Table 2 Data included in the enriched OOM event log

[0043] In Table 2, fcontainer_name, fpod_name, and fnamespace are extended fields.

[0044] Step 305: Report the memory overflow event log.

[0045] Step 306, determining potential malicious behavior according to the memory overflow event log and preset alarm rules; outputting the potential malicious behavior for alarm.

[0046] In this embodiment, the memory overflow event log can be further analyzed and processed. Specifically, the preset alarm rules can be combined to determine potential malicious behavior. For example, the preset alarm rule is that the pod under the specified user namespace triggered the OOM event. The user namespace is a very important resource in the kubernetes system. Its main function is to achieve resource isolation of multiple environments or resource isolation of multiple tenants. After the cluster is started, several user namespaces will be created by default. If it is analyzed in the memory overflow time log that the pod under the specified user namespace triggered the OOM event, it is considered that the behavior is a resource abuse or an attack.

[0047] Furthermore, big data analysis platforms, such as the Flink streaming computing engine, can be used for analysis to identify pods, containers, and processes that frequently trigger OOM, generate alarms for potential malicious behaviors (such as resource abuse, DDOS attacks, etc.), and notify operation and maintenance personnel via email, SMS, or enterprise communication tools.

[0048] Step 307: Analyze the memory overflow event log to determine the container identifier for which the number of memory overflow events exceeds a preset value; and adjust the configuration information corresponding to the container identifier.

[0049] In addition, you can also analyze the memory overflow event log to determine the container ID for which the number of memory overflow events exceeds the preset value. For example, if Pod B under user namespace A has three OOM events within an hour, it is considered that the memory resource limit of Pod B under user namespace A is small. At this time, you can call Kubernetes API to increase the memory resource limit of Pod B under user namespace A to 1.5 times the original, or call Kubernetes API to set the node affinity of Pod B under user namespace A to schedule it to a node with abundant memory resources.

[0050] Step 308: Analyze the memory overflow event log to determine the memory overflow information of each node and container, the parent-child relationship of the process that triggered the memory overflow event, and the parent-child relationship of the killed process; and visualize the memory overflow information and the parent-child relationship.

[0051] In addition, the memory overflow event log can be analyzed to determine the memory overflow information of each node and container, the parent-child relationship of the process that triggered the memory overflow event, and the parent-child relationship of the killed process. Specifically, it can be determined when and why the memory overflow event occurred for each node or each container. Furthermore, the above memory overflow information and parent-child relationship can be displayed through a visualization tool. Furthermore, when displaying, the historical memory overflow information and real-time memory overflow information of each node and container can also be displayed.

[0052] In some optional implementations of this embodiment, the memory overflow information of a single node and container can be displayed in real time through a dashboard; based on the historical memory overflow information within a preset time period, the memory overflow event triggering trend of a single node and container can be determined. In this way, operation and maintenance personnel can make decisions.

[0053] The memory overflow event processing method provided by the above-mentioned embodiment of the present disclosure adopts the eBPF kprobe technology to mount the kernel function oom_kill_process, which is non-invasive to the business and can capture OOM events in real time with extremely low performance overhead. Compared with traditional monitoring methods, this solution can obtain OOM event information in real time, efficiently and accurately, avoiding delays and omissions. In addition, the detailed context information of the process that triggers OOM and the process killed by OOM can be accurately captured in the kernel state, including process ID, path, container ID, memory used by the process, etc., and then the container and Pod metadata are enriched by the user state to ensure the integrity and accuracy of the OOM event data. This provides complete basic data for subsequent analysis and response. By enriching the OOM event log, the event is associated with metadata such as containers, Pods, and namespaces, and the close association of OOM events with businesses in the cloud native environment is achieved. The relevance of this contextual information enables operation and maintenance personnel to quickly identify the business and container of the problem and improve the efficiency of problem location. By automating resource adjustment or node migration for businesses that frequently experience OOM events, service interruptions caused by resource contention are avoided. This automated response capability significantly reduces the manual intervention of operation and maintenance personnel, and improves system stability and business continuity. By analyzing real-time OOM events through the big data analysis platform and combining it with preset alarm rules, it can detect and identify abnormal situations in a timely manner, and notify relevant personnel through multiple channels, realizing a flexible alarm mechanism to ensure that the operation and maintenance team can respond quickly. It supports real-time display and historical query of OOM events, and provides powerful monitoring and analysis capabilities. Operation and maintenance personnel can use intuitive dashboards to view system status in real time and identify resource hotspots. At the same time, they can further optimize resource allocation and strategies through historical data analysis.

[0054] Further references Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a memory overflow event processing device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0055] like Figure 4 As shown, the memory overflow event processing device 400 of this embodiment includes: an event capturing unit 401 , a data collecting unit 402 , a data enriching unit 403 and a log reporting unit 404 .

[0056] The event capture unit 401 is configured to capture the memory overflow event in real time in the kernel state through a detection function pre-mounted in a preset function.

[0057] The data collection unit 402 is configured to collect data related to the memory overflow event in response to capturing the memory overflow event, and send the related data to the user state.

[0058] The data enrichment unit 403 is configured to perform enrichment processing on the relevant data of the memory overflow event in the user state to obtain a memory overflow event log.

[0059] The log reporting unit 404 is configured to report the memory overflow event log.

[0060] In addition, in the technical solution of the present application, an electronic device is also proposed.

[0061] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0062] like Figure 5 As shown, the electronic device may include a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501, wherein the processor 501 and the memory 502 communicate with each other via the bus 503. The steps of implementing the above method when the processor 501 executes the computer program include, for example: in kernel state, capturing memory overflow events in real time through a detection function pre-mounted in a preset function; in response to capturing a memory overflow event, collecting relevant data of the memory overflow event and sending the relevant data to a user state; in the user state, enriching the relevant data of the memory overflow event to obtain a memory overflow event log; and reporting the memory overflow event log.

[0063] In addition, in one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: in kernel state, capturing memory overflow events in real time by a detection function pre-mounted in a preset function; in response to capturing a memory overflow event, collecting relevant data of the memory overflow event, and sending the relevant data to a user state; in the user state, enriching the relevant data of the memory overflow event to obtain a memory overflow event log; and reporting the memory overflow event log.

[0064] To sum up, in the technical solution disclosed in the present invention, the relevant data of memory overflow time can be collected by mounting a detection function on a preset function in the kernel state, and the relevant data can be sent to the user state, and the relevant data can be enriched in the user state to obtain a memory overflow event log, and the memory overflow event log can be further reported, so that the information of memory overflow events can be obtained in real time, efficiently and accurately, avoiding delays and omissions.

[0065] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A method for handling a memory overflow event, the method comprising: In kernel state, memory overflow events are captured in real time through the detection function pre-mounted in the preset function; In response to capturing a memory overflow event, collecting data related to the memory overflow event, and sending the data related to the user state; In the user state, enriching the relevant data of the memory overflow event to obtain a memory overflow event log; Report the memory overflow event log.

2. The method according to claim 1, wherein: The method further comprises: A probe is inserted at the preset function, and the detection function is mounted at the probe.

3. The method according to claim 1, wherein: The enrichment processing of the relevant data of the memory overflow event includes: determining a container identification from the associated data; Determining, according to the container identifier, container metadata corresponding to the container identifier; The relevant data is enriched according to the container metadata.

4. The method according to claim 1, wherein: The method further comprises: Determine potential malicious behavior based on the memory overflow event log and preset alarm rules; The potential malicious behavior is output for warning.

5. The method according to claim 1, wherein: The method further comprises: Analyze the memory overflow event log to determine the container identifiers for which the number of memory overflow events that occurred exceeds a preset value; The configuration information corresponding to the container identifier is adjusted.

6. The method according to claim 1, wherein: The method further comprises: Analyze the memory overflow event log to determine the memory overflow information of each node and container, the parent-child relationship of the process that triggered the memory overflow event, and the parent-child relationship of the killed process; The memory overflow information and the parent-child relationship are displayed visually.

7. The method according to claim 6, wherein: The visual display of the memory overflow information and the parent-child relationship includes: The dashboard displays the memory overflow information of a single node and container in real time. Based on the historical memory overflow information within a preset time period, the memory overflow event triggering trend of a single node and container is determined.

8. A memory overflow event processing device, the device comprising: The event capture unit is configured to capture the memory overflow event in real time in the kernel state through a detection function pre-mounted in a preset function; A data collection unit, configured to collect relevant data of the memory overflow event in response to capturing the memory overflow event, and send the relevant data to the user state; A data enrichment unit is configured to perform enrichment processing on the relevant data of the memory overflow event in a user state to obtain a memory overflow event log; The log reporting unit is configured to report the memory overflow event log.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the memory overflow event processing method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the memory overflow event processing method as described in any one of claims 1 to 7 is implemented.

Images