AI training container monitoring method and device with memory abnormity early warning function and medium

Abstract

The invention discloses an AI training container monitoring method and device with a memory abnormity early warning function and a medium. The method comprises the steps that 1, a memory limiting amount and a container stability strategy are specified in configuration of a target AI training container; step 2, obtaining a container memory limiting amount and a container stability strategy through configuration of a target AI training container, and persisting the container memory limiting amount and the container stability strategy; 3, continuously obtaining the actual memory usage amount of the target AI training container, judging whether the actual memory usage amount of the container is greater than the persistent container memory limit amount, and if so, triggering an alarm information notification; and 4, when the user mode service accepts the notification, operating the target AI training container according to the container stability strategy. The method has a memory exception early warning function, and the training stability and reliability of the AI container are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AI model training, and particularly relates to an AI training container monitoring method and device with memory exception early warning function and a medium. BACKGROUND

[0002] In the AI training scene, containerized deployment is widely used to improve resource isolation and system flexibility. Through mechanisms such as cgroup, the resources of the container can be limited in a fine-grained manner, and memory limitation is a common means to ensure the overall availability of the system. However, when the container sets a too small memory limit value and does not configure a suitable mitigation strategy, it is easy to cause memory overflow (OOM) in the AI model training process, trigger the kernel OOM killer to forcibly terminate the process, and cause problems such as training interruption and model unsaved, which seriously affects the stability and reproducibility of the training task.

[0003] However, in the AI training container, due to the strong transient nature of memory usage, the AI training process (such as large batch size, gradient accumulation, and memory / hybrid memory scheduling) will have sudden memory consumption. Traditional user-mode sampling (such as cAdvisor and top) often has a sampling period of seconds, which can easily miss the millisecond-level peak, resulting in no early warning before OOM. Moreover, the current AI training container lacks the linkage of exception detection and stability strategies, and can only achieve the alarm function (such as the monitoring and alarm combination system of Prometheus+AlertManager), and cannot quickly link with the container runtime (such as pause and restart), so that even if an exception is detected, it will still be forcibly terminated by the OOM Killer. Therefore, the current AI training container is actually difficult to capture the memory exception state in real time, and even more difficult to execute the corresponding stability strategy.

[0004] In the prior art, although the Linux cgroup (control group) mechanism and the like can perform fine-grained limitation on the memory resources of a container, the mechanism only provides static threshold constraints and simple over-limit processing. When the memory usage of the container exceeds the limit, the OOM Killer is directly triggered to forcibly terminate the process, and there is a lack of differentiated management strategies for specific application scenarios. In the AI training scenario, the sensitivity, fault tolerance, and recovery mechanism of different training tasks to memory are obviously different. For example, in the large-scale model training process, short-term expansion or delayed recycling may be needed during the memory peak period, while small-scale experimental tasks are more suitable for direct interruption or restart. However, the existing cgroup limitation mechanism cannot make dynamic decisions according to the business attributes, training stages, or strategy configurations of different containers, and also lacks linkage capabilities with the container runtime. As a result, when memory exceptions occur, only a "one-size-fits-all" coarse-grained disposal method can be used, which cannot achieve policy-based decisions and robust management for different AI training containers, thereby directly leading to a decrease in the stability and reproducibility of training tasks and increasing the risk of model loss or training failure caused by OOM interruption. SUMMARY

[0005] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides an AI training container monitoring method, device and medium with a memory exception early warning function, which can efficiently and low-costly obtain the actual memory usage of an AI training container and timely trigger container stability operations, effectively avoiding the pure reliance on OOM Killer and improving the training stability and reliability of AI containers.

[0006] To solve the above technical problems, the technical solution provided by the present application is as follows: An AI training container monitoring method with a memory exception early warning function, comprising the following steps: Step 1: specifying a memory limit and a container stability strategy in the configuration of a target AI training container; Step 2: obtaining the memory limit and the container stability strategy of the target AI training container through the configuration of the target AI training container, and persisting the memory limit and the container stability strategy of the target AI training container; Step 3: continuously obtaining the actual memory usage of the target AI training container, and judging whether the actual memory usage of the target AI training container is greater than the persisted container memory limit. If yes, an alarm notification is triggered; Step 4: when a user-mode service accepts the alarm notification, operating the target AI training container according to the container stability strategy.

[0007] Further, in step 1, the container stability strategy is an operation strategy for the AI training container when the actual memory usage of the AI training container is greater than the memory limit of the AI training container, and the operation strategy includes a restart operation, a pause operation, a delete operation, a continue operation, a terminate operation, and no operation. The configuration of the AI training container includes the configuration of the AI training container created by the AI training container runtime software and the configuration of the AI training container created by the AI training container orchestration software.

[0008] Further, in step 1, a YAML file is created using the VIM command of the Linux operating system to define the configuration of the Pod, and the YAML file contains the necessary information of the Pod, including the name of the Pod, the image of the container, and the container configuration information. The YAML file of the Pod is created into a Pod by the kubectl apply command.

[0009] Further, in step 2, a storage directory is created on a local disk, and a meta-information file is created in the storage directory, which contains the memory limit of the AI training container and the stability strategy. The meta-information file also includes the container process ID, container ID, and container name.

[0010] Further, step 3 includes: Step 3.1: Deploy eBPF program in kernel mode on the host machine, and mount to the kernel memory page to capture memory allocation or release events; Step 3.2: Maintain the mapping relationship between the AI training container PID and the actual memory usage value in the kernel eBPF map; Step 3.3: Filter the memory events of the target AI training container, and update the memory usage value of each AI training container in the eBPF kernel map using the mapping relationship; Step 3.4: If it is monitored that the memory usage of the AI training container exceeds the container memory limit, send an alarm notification to the user mode service.

[0011] Further, in step 3.3, the memory events of the target AI training container are double-filtered according to the cgroup id and the AI training container PID, and the memory usage value of each AI training container in the eBPF kernel map is updated in an incremental manner using the mapping relationship. When updating, if it is a memory allocation event, the corresponding byte number is added to the original value, and if it is a memory release event, the corresponding byte number is reduced from the original value, forming a dynamic approximate estimation of the actual memory usage.

[0012] Further, step 4 includes: Step 4.1: the user state service triggers an alarm notification event when the actual memory usage of the target AI training container is greater than the container memory limit; Step 4.2: when the alarm notification event is monitored, read the event data information, which includes the container process ID and the container ID, and query the corresponding persistent container stability policy according to the event data information; Step 4.3: execute the persistent container stability policy obtained by the query.

[0013] Further, in step 4.1, the user state daemon process establishes a zero-copy communication channel with the kernel perf ringbuffer through the perf_event interface to monitor the alarm notification event, and triggers the alarm notification event only when the actual memory usage of the target AI training container is greater than the container memory limit and the cooling period meets the condition, thereby realizing the alarm anti-jitter mechanism. A computer device, comprising a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the method as described above.

[0014] A computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the method as described above.

[0015] Compared with the prior art, the present application has the following advantages: 1. The present application can avoid the problems of large delay, coarse granularity and high resource overhead in traditional user state monitoring by realizing millisecond-level, low-overhead monitoring of AI training container memory usage in kernel space with the help of eBPF technology. By comparing the container memory usage with the preset threshold at a high frequency, timely warning can be given before OOM occurs, and the steady state guarantee operation can be performed by the user state policy engine. Compared with the traditional direct cgroup limit setting, the present application can provide more flexible training process control ability, reduce the risk of training interruption and result loss caused by accidental OOM, and improve the stability and reproducibility of AI training tasks, as well as the training stability and reliability of AI containers.

[0016] 2. The present application can perceive the transient changes of memory at a millisecond level without configuring fixed memory limits through cgroup, complete abnormality detection in a low-delay and low-overhead manner, realize a millisecond-level granularity monitoring mechanism for real-time capture of container memory state, and provide differentiated running strategies based on the characteristics and training needs of different containers, so as to realize dynamic governance of memory abnormalities, and at the same time have the linkage ability with the container runtime, so as to realize early warning and policy response of memory abnormalities. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is the implementation flow diagram of the AI training container monitoring method with memory exception early warning function of the embodiment.

[0018] Figure 2 is the complete flow diagram of the implementation of AI training container monitoring early warning and strategy execution operation. DETAILED DESCRIPTION

[0019] The present application is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present application is not limited thereby.

[0020] As shown in Figure 1 , the steps of the AI training container monitoring method with memory exception early warning function of the embodiment include: Step 1: Specify the memory limit and container stability strategy in the configuration of the target AI training container.

[0021] Specifically, the memory early warning threshold (such as the historical maximum memory limit) and the container stability strategy (such as performing restart, pause or no operation when the threshold is reached) are specified in the configuration of the target AI training container, the default cgroup memory limit of the container is cancelled, and OOM is prevented.

[0022] Specifically, the container stability strategy is the operation strategy performed on the target AI training container when the actual memory usage of the target AI training container is greater than the container memory limit. The operation strategy includes: restart operation, pause operation, delete operation, continue running operation, terminate running operation and no operation, etc. The configuration of the AI training container includes: the configuration of the container runtime software to create the AI training container, the configuration of the container orchestration software to create the AI training container, etc.

[0023] In a specific application embodiment, the container orchestration software includes Kubernetes, etc. The container orchestration software can use Kubernetes, etc., and the container runtime software uses docker, runc, cri-o, etc. The configuration of creating an AI training container in the container orchestration software Kubernetes can use the VIM command of the Linux operating system to create a YAML (YAML Ain't Markup Language) file to define the configuration of a Pod, and the Pod encapsulates one or more containers. The YAML file contains all the necessary information of the Pod, such as the name of the Pod, the image of the container, the configuration information of the container, etc. The memory limit of the AI training container is specified as 2G in the configuration information of the AI training container, and the stability policy is specified, such as: no operation. The configuration YAML file of the Pod is created into a Pod through the kubectl apply command in Kubernetes. At the same time, the default cgroup memory limit of the container is cancelled to prevent OOM.

[0024] Step 2: Obtain the memory limit of the target AI training container and the container stability policy through the configuration of the target AI training container, and persist the memory limit of the target AI training container and the container stability policy.

[0025] Specifically, the persistence also includes container process ID, container ID, container name, etc., which can be persisted in local disk, network storage, database, other persistent storage media, etc.

[0026] In this embodiment, the memory limit of the container and the container stability policy are obtained through the configuration of the target AI training container, a storage directory storage01 is created in the local disk, a meta information file is created in the storage directory storage01, and the meta information file contains memory limit, stability policy, container process ID, container ID, container name, etc.

[0027] Step 3: Continuously obtain the actual memory usage of the target AI training container, and judge whether the actual memory usage of the target AI training container is greater than the persisted container memory limit. If yes, trigger an alarm notification.

[0028] In this embodiment, by deploying eBPF programs in the host kernel state, the programs contain running parameters such as container memory limit parameters, container IDs, and container process IDs, and under the premise of as low overhead as possible, event-drivenly capture memory allocation / release related kernel events, double-filter by container process ID and cgroup_id, and maintain the "approximate actual memory usage" (estimated based on page allocation or release events) of each container in the kernel map in an incremental manner. When the estimated usage of a certain container exceeds the persistence threshold, the alarm event (including container identification, current estimated value, timestamp, triggering process, etc.) is sent to the user state daemon through the bpf_perf_event_output() function, triggering policy execution.

[0029] In this embodiment, the specific steps of step 3 include: Step 3.1: Deploy eBPF (Extended Berkeley Packet Filter) programs in the kernel state of the host, and mount them to the kernel memory page to capture memory allocation or release events; This embodiment loads eBPF programs in the container host kernel and mounts the programs to the memory management related tracepoint events provided by the Linux kernel, such as mm_page_alloc and mm_page_free. When the kernel performs memory page allocation or release operations, the tracepoint will automatically trigger the corresponding eBPF probe function. In this way, the memory behavior of the container during operation can be captured in real time in an event-driven manner without modifying the kernel source code. By capturing memory operations in an event-driven manner, the traditional polling / proc or cgroup file system method can be avoided, significantly reducing performance overhead and improving real-time monitoring and accuracy.

[0030] Step 3.2: Maintain the mapping relationship between AI training container PID (process ID) and actual memory usage value in the kernel eBPF map.

[0031] After capturing the memory event, the eBPF program calls the auxiliary function bpf_get_current_pid_tgid() to obtain the process identifier of the triggering event, and maintains the corresponding memory usage value in the kernel eBPF map using the container cgroup id and container process ID as the composite key value. The value in the map is the "actual memory usage" accumulated by the container process, which is calculated based on the incremental update of the page allocation or release size reported by the kernel. By maintaining the mapping relationship between (cgroup id, container process ID) and memory usage in the kernel map, this embodiment can track the memory usage in the container with fine-grained granularity and avoid frequent user-kernel state switching, thereby improving the overall efficiency of the system.

[0032] Step 3.3: Filter the memory events of the target AI training container, and update the memory usage values of each container in the eBPF kernel map using the mapping relationship.

[0033] Before updating the map, the eBPF program calls bpf_get_current_cgroup_id() to obtain the process cgroup id of the triggering event, and compares it with the user-specified target container cgroup id. Only when they match will the map be updated.

[0034] Further, the memory events of the target AI training container can be double-filtered according to the cgroup id and AI training container PID, and the memory usage values of each AI training container in the eBPF kernel map can be updated in an incremental manner using the mapping relationship. That is, the container process ID is filtered to achieve more fine-grained monitoring. When updating, if it is a memory allocation event, the corresponding byte number is added to the original value; if it is a memory release event, the corresponding byte number is subtracted from the original value, thereby forming a dynamic approximation of the actual memory usage. By using the double-filtering mechanism (cgroup id + PID), this embodiment can ensure that only the memory events of the target container are counted, avoiding interference from other processes or containers, and the incremental update mechanism can update in real time when an event is triggered, significantly improving real-time performance and accuracy compared to traditional periodic scanning.

[0035] Step 3.4: If the memory usage of the target AI training container exceeds the container memory limit, send an alarm notification to the user-mode service.

[0036] When the cumulative memory usage of the target AI training container in the kernel map exceeds the persistent container memory limit, the eBPF program constructs event data containing the container process ID, container ID, cgroup id, current usage value, memory limit, timestamp, and other key information, and calls bpf_perf_event_output() to write the event data to the perf ring buffer. The user-space daemon receives the alarm event through the perf buffer and performs corresponding processing on the target AI training container according to the persistent container stability policy (such as restart, pause, delete, or no operation).

[0037] Further, the embodiment adopts an asynchronous reporting mechanism, uses the perf ring buffer to achieve efficient communication between the kernel state and the user state, and avoids blocking the kernel execution path. At the same time, an alarm anti-jitter mechanism is adopted, which records the last reported usage in the map and triggers reporting only when the memory growth exceeds the preset container memory limit, thereby avoiding alarm storms caused by frequent fluctuations. The notification event contains multi-dimensional information such as container ID, PID, and timestamp, which facilitates fast positioning and policy execution by the user-space service, forming a complete strategy from detection to alarm to governance.

[0038] In a specific application embodiment, for example, Figure 2As shown, the complete process of monitoring the actual memory usage of the target AI training container and triggering the alarm notification is as follows: loading and deploying the eBPF program in the host kernel, the program contains running parameters such as memory limit parameters of the target AI training container, container ID and container process ID, and mounting the program to the memory management tracepoint provided by the kernel mm_page_alloc and mm_page_free, when the process in the target AI training container triggers memory page allocation or release, the corresponding eBPF probe function is automatically called to realize the event-driven capture of the memory behavior of the container. Subsequently, the eBPF program uses bpf_get_current_pid_tgid() to obtain the process identifier of the triggering event, and uses the cgroup id of the container and the container process ID as a composite key to maintain the corresponding memory usage value in the kernel eBPF map, said value is updated incrementally according to the allocation and release events, forming the cumulative memory usage of each process in the container. During the updating process, the program obtains the cgroup id of the event process by bpf_get_current_cgroup_id(), and compares it with the cgroup id of the target container, and only when they match will the map update operation be performed, combined with the container process ID fine filtering, to ensure that the statistical result only reflects the actual memory usage of the target container, and the updating method is event-driven incremental calculation, thereby improving the real-time and accuracy of monitoring. When the cumulative memory usage of the container is detected to exceed the threshold value stored persistently, the eBPF program constructs an alarm event containing context information such as the container process ID, container ID, cgroup id, timestamp, memory limit and current memory usage, and writes it into the perf ring buffer through bpf_perf_event_output(), which is received by the user space daemon and triggers the corresponding stability policy execution, for example, according to the preset configuration to select "no operation", thereby completing the memory usage monitoring and policy management control of the container.

[0039] Step 4: When the user space service receives the alarm notification, operate the target AI training container according to the container stability policy.

[0040] In this embodiment, the user state daemon process establishes a zero-copy communication channel with the kernel perf ring buffer through the perf_event interface, which can achieve low-latency and low-overhead real-time monitoring of memory alarm events; by analyzing event data, the daemon process dynamically queries the container stability policy in the persistent storage, and performs restart, pause, termination or continue operation according to the policy of different containers. Through the above method, fine-grained memory management and fast linkage with container runtime can be realized, thereby improving the stability and resource management efficiency of AI training tasks. Further, an alarm anti-jitter mechanism can be introduced to ensure that container management operations are only performed when the memory usage exceeds the limit and the cooling period meets the conditions, thereby avoiding misoperation or frequent restarts and ensuring the stability and accuracy of container management.

[0041] Step 4.1: The user state service listens to the alarm notification event triggered when the actual memory usage of the target AI training container is greater than the container memory limit.

[0042] In this embodiment, the user state daemon process establishes a communication channel with the kernel perf ring buffer based on the perf_event interface provided by Linux, and continuously monitors the memory alarm events reported by the eBPF program. Specifically, the user state service creates a perf fd by calling perf_event_open() in the initialization stage, and maps the perf buffer to the user space through mmap(), so that it can directly read the event data written by the kernel in a zero-copy manner, avoiding the overhead caused by frequent kernel traps in the traditional system call mode. By using the event-driven and zero-copy reading mechanism, the user state can obtain kernel alarm information in real time under the premise of low latency and low overhead.

[0043] Step 4.2: When the alarm notification event is monitored, read the event data information, which includes container process ID, container ID, etc., and query the corresponding persistent container stability policy according to the event data information.

[0044] When a new event record appears in the perf buffer, the user-mode daemon parses the data structure reported by eBPF, extracts the container process ID, container ID, memory usage, timestamp, and other key context information. Subsequently, the daemon accesses the local persistent storage (such as etcd) using the container process ID and container ID as query keys to retrieve the stability policy corresponding to the container. This policy can include but is not limited to: continue running, do nothing, restart the container, suspend the container, terminate the container, and other operations. By using the dynamic mapping and querying method of event data and persistent policies, this embodiment can decouple the monitoring logic and governance strategy, allowing the strategy to be configured differently based on the business importance or running environment differences of different containers, thereby improving the flexibility and scalability of the system.

[0045] Step 4.3: Execute the query to obtain the persistent container stability policy.

[0046] In a specific application embodiment, as shown in Figure 2 The user-mode service listens to alarm events, and the detailed steps for operating the target AI training container according to the container stability policy are as follows: The user-mode daemon first maps the perf ring buffer to the user space by calling perf_event_open() and combining mmap(), to listen to memory alarm events reported by eBPF programs in a event-driven and zero-copy manner. When a new event record is detected to be written through event-driven detection (such as poll or epoll), the daemon immediately parses the data structure, extracts the container process ID, container ID, memory usage, timestamp, and other context information, and accesses the local persistent storage (such as etcd) using the container identifier as a query key to retrieve the corresponding stability policy, such as continue running, do nothing, restart, suspend, or terminate the container. Subsequently, the daemon calls the container runtime interface (such as the Docker Engine API or containerd gRPC interface) based on the query result to perform the corresponding control operation on the target container. Further, an alarm debounce mechanism is introduced in the policy execution link, that is, the memory usage state of the container is checked again before actual operation, and it is judged whether the time since the last policy execution exceeds the preset cooling period, and only when the conditions are met, the restart, suspend, or terminate operation is performed, thereby effectively avoiding the phenomenon of false triggering or frequent restarts caused by short-term fluctuations. Through the above-mentioned method, real-time perception, policy-based decision-making, and robust governance of the user-mode to the target container are realized, which can ensure the integrity and stability of the monitoring-alarm-governance process.

[0047] Compared with a traditional mode of relying on user state periodic polling or log collection, the application can perceive millisecond-level transient changes in memory during the running of an AI training container without cgroup configuration of fixed memory limits, complete abnormality detection in a low-delay and low-overhead manner, realize a monitoring mechanism for real-time capture of container memory states at a millisecond-level granularity, and provide differentiated running strategies based on the characteristics and training needs of different containers to realize dynamic governance of memory abnormalities. At the same time, the application also has linkage capability with container runtime, and can automatically govern containers according to configured stability strategies (such as pausing, restarting, terminating or ignoring) when detecting memory usage abnormalities, thereby realizing early warning and strategic response of memory abnormalities. Through the above-mentioned mode, the actual memory usage of an AI training container can be obtained at a millisecond level and with low overhead without cgroup configuration of fixed memory limits, and timely warning or triggering of container stability operations can be realized when a historical limit or a strategy setting threshold is about to be reached, which can effectively avoid the problems of training task interruption and model loss caused by pure reliance on OOMKiller, improve the stability and reproducibility of AI training tasks, and improve the training stability and reliability of AI containers.

[0048] The embodiment further provides a computer device, including a processor and a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program to execute the method.

[0049] It can be understood that the above-mentioned method of the embodiment can be executed by a single device, such as a computer or a server, and can also be applied to a distributed scenario and completed by multiple devices in cooperation. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above-mentioned method of the embodiment, and the multiple devices interact to complete the above-mentioned method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute related programs to implement the above-mentioned method of the embodiment. The memory can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above-mentioned method of the embodiment is implemented by software or firmware, the related program codes are saved in the memory and executed by the processor.

[0050] The embodiment further provides a computer readable storage medium storing a computer program, the computer program is executed by a processor to implement the above-mentioned method.

[0051] Those skilled in the art will appreciate that the above-described embodiments of the present application can be embodied as a method, system, or computer program product. Accordingly, the present application can be embodied in hardware alone, software alone, or in combination with software and hardware. Moreover, the present application can be embodied in a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer usable program code embodied thereon. The present application is described in the general context of method, apparatus (system), and computer program product, according to one embodiment of the application. The computer program product can be implemented in a local memory of a computing device or in a memory of a computing device accessible via a network, such as the Internet or World Wide Web. Generally, computer program product comprises computer usable program code, which is a physical tangible entity that can be used directly, or indirectly, by a computing device to Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart Figure 1 The computer program product can also be loaded onto a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer implemented process such that the instructions which execute on the computer or other programmable device provide steps for implementing the functions specified in the flowchart

[0052] The foregoing is considered as illustrative only of the principles of the application. Further, since numerous modifications and changes will readily occur to those skilled in the art, it is not desired to limit the application to the exact construction and operation shown. Therefore, accordingly, all such variations are intended to be included within the scope of present application as defined in the following claims, the novel features of which are to be understood as being applicable to each defined aspect.

Claims

1. A method for monitoring AI training containers with memory anomaly warning function, characterized in that the steps include: include: Step 1: Specify the memory limit and container stability policy in the configuration of the target AI training container; Step 2: Obtain the memory limit and stability policy of the target AI training container through the configuration of the target AI training container, and persist the memory limit and stability policy of the target AI training container; Step 3: Continuously acquire the actual memory usage of the target AI training container, and determine whether the actual memory usage of the target AI training container is greater than the persistent container memory limit. If so, trigger an alarm notification. Step 4: When the user-space service receives the alarm notification, it operates the target AI training container according to the container stability strategy.

2. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 1, characterized in that, In step 1, the container stability strategy is the operation strategy for the AI ​​training container when the actual memory usage of the AI ​​training container exceeds the memory limit of the AI ​​training container. The operation strategy includes: restart operation, pause operation, delete operation, continue running operation, terminate running operation, and no operation. The configuration of the AI ​​training container includes: the configuration of the AI ​​training container runtime software to create the AI ​​training container and the configuration of the AI ​​training container orchestration software to create the AI ​​training container.

3. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 1, characterized in that, In step 1, a YAML file is created using the VIM command of the Linux operating system to define the configuration of the Pod. The YAML file contains the necessary information of the Pod, including the Pod name, the container image, and the container configuration information. The Pod is created as a Pod using the kubectl apply command.

4. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 1, characterized in that, In step 2, a storage directory is created on the local disk, and a metadata file is created in the storage directory. The metadata file contains the memory limit and stability policy of the AI ​​training container. The metadata file also includes the container process ID, container ID, and container name.

5. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 1, characterized in that, Step 3 includes: Step 3.1: Deploy the eBPF program in kernel mode on the host machine and mount it to a kernel memory page to capture memory allocation or deallocation events; Step 3.2: Maintain the mapping relationship between the AI ​​training container PID and the actual memory usage value in the kernel eBPF map; Step 3.3: Filter the memory events of the target AI training container and update the memory usage value of each AI training container in the eBPF kernel map using the mapping relationship; Step 3.4: If the AI ​​training container memory usage exceeds the container memory limit, an alarm notification is sent to the user-space service.

6. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 5, characterized in that, In step 3.3, the memory events of the target AI training container are filtered twice based on the cgroup id and the AI ​​training container PID. The memory usage value of each AI training container in the eBPF kernel map is updated incrementally using the mapping relationship. During the update, if it is a memory allocation event, the corresponding number of bytes is added to the original value; if it is a memory release event, the corresponding number of bytes is reduced to the original value, forming a dynamic approximate estimate of the actual memory usage.

7. The AI ​​training container monitoring method with memory anomaly early warning function according to any one of claims 1 to 6, characterized in that, Step 4 includes: Step 4.1: The user-space service listens for alarm notification events triggered when the actual memory usage of the target AI training container exceeds the container's memory limit; Step 4.2: When an alarm notification event is detected, read the event data information, which includes: container process ID and container ID. Query the corresponding persistent container stability policy based on the event data information. Step 4.3: Execute the persistent container stability strategy obtained from the query.

8. The AI ​​training container monitoring method with memory anomaly early warning function according to claim 7, characterized in that, In step 4.1, the user-space daemon establishes a zero-copy communication channel with the kernel perf ring buffer through the perf_event interface to listen for alarm notification events. The alarm notification event is only triggered when the actual memory usage of the target AI training container exceeds the container memory limit and the cooldown period meets the conditions, thus implementing an alarm debouncing mechanism.

9. A computer device comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images