Method and device for storing training data of writable layer of AI training container and medium

Abstract

The invention discloses an AI training container writable layer training data storage method and device and a medium, and the method comprises the steps: 1, continuously sensing the internal training behaviors of a container, judging whether a storage triggering condition is met or not, and sending a notification event to a user mode service if the storage triggering condition is met; step 2, when the user mode service receives a notification event, storing the writable layer file to obtain a container writable layer archive, extracting container feature information by creating an archive structure containing a writable layer, a read-only layer and meta-information, writing the container feature information into a meta-information file, and storing the container feature information into the user mode service; and storing the writable layer file of the current container and the covered read-only layer file to local or remote persistent storage. According to the method, triggering of the writable layer data backup strategy can be driven through the training behavior, key node data loss is avoided, and the data retention capacity and evaluation and recovery capacity of AI model training are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of AI model training, and in particular to an AI training container writable layer training data saving method, device and medium. BACKGROUND

[0002] AI model training usually adopts a container runtime environment (such as a Docker or Kubernetes container) to provide a unified and reproducible dependency environment. The container adopts a layered file system structure, in which the writable layer (UpperDir) is used to record dynamic data generated during container runtime, such as intermediate results generated during the training process, model parameter snapshots, temporary logs, etc. The writable layer is temporary and will be emptied after the container is deleted, which will cause the loss of key training data.

[0003] In the prior art, for the backup mechanism of the container writable layer, the following two types of methods are usually adopted: One type of method is to trigger a backup operation at fixed time intervals. However, the key nodes of the model training process are often dynamic and unpredictable, such as when the loss function (Loss) experiences a significant drop, the model enters the convergence stage, or a key checkpoint is generated. If the timing period is too long, these key data may be missed; if the timing period is too short, the storage and I / O burden will be significantly increased, and even the training performance will be affected.

[0004] Another type of method is to uniformly trigger data saving at the end of the container life cycle. However, if the training task is terminated prematurely due to system restart, resource exhaustion (OOM), abnormal exit, etc., the intermediate process data cannot be saved, and the training results cannot be recovered.

[0005] Therefore, the existing writable layer backup mechanism does not have fine-grained perception ability for the dynamic behavior of the training process, and it is difficult to accurately capture key behavior stages. Therefore, it is necessary to provide an AI training container writable layer training data saving method to avoid the loss of key data or intermediate process data during the training process and improve the data retention ability and evaluation and recovery ability of AI model training. SUMMARY

[0006] The technical problem to be solved by the present application is that, in view of the technical problems existing in the prior art, the present application provides an AI training container writable layer training data saving method, device and medium, which can trigger the backup strategy of the writable layer data based on the training behavior, avoid the loss of key data or intermediate process data at key nodes, and improve the data retention ability and evaluation and recovery ability of AI model training.

[0007] To solve the above technical problems, the technical solution provided by the present application is: An AI training container writable layer training data saving method, steps include: Step 1: In the AI model training process, continuously perceive the training behavior inside the container, judge whether the saving trigger condition is met, and if so, send a notification event to the user state service; Step 2: When the user state service receives the notification event, save the writable layer file to get the container writable layer archive, wherein by creating an archive structure containing the writable layer, the read-only layer and the meta information, after extracting the container feature information and writing it into the meta information file, the current container writable layer file and the covered read-only layer file are saved to the local or remote persistent storage.

[0008] Further, step 1 includes: Step 1.1: Deploy eBPF program on host machine, mount to kernel function related to file system, said kernel function includes monitoring file creation operation vfs_create, monitoring file opening operation do_sys_open; Step 1.2: The eBPF program filters the system call behavior in the target container by identifying the container environment; Step 1.3: The eBPF program maintains a map structure in the kernel, and triggers the eBPF program every time a new file is created in the container, and the eBPF program records the number of file creations in the container; Step 1.4: If the number of file creations reaches a preset threshold, trigger a user space event, send a notification event to the user space through perf_event, said notification event carries cgroup_id or container identification information.

[0009] Further, in step S1.2, the eBPF program identifies the container environment by bpf_get_current_cgroup_id() and namespace check, and only counts the system call behavior occurring in the specified container, wherein bpf_get_current_cgroup_id() is used to identify the cgroup ID of the container, and namespace check is used to verify the isolation environment. In step S1.3, the number of file creations for each container is saved by a global shared hash table BPF_HASH or a single CPU hash table BPF_PERCPU_HASH map in the kernel.

[0010] Further, in step 2, creating an archive structure containing the writable layer, the read-only layer and the meta information includes: Step 2.1: The user state service listens to the perf_event notification event; Step 2.2: After the user state service receives the perf_event notification event, call the container runtime tool to get the container writable layer and read-only layer path; Step 2.3: generating a unique storage directory path according to the container ID and the timestamp; Step 2.4: creating an archive structure containing a writable layer, a read-only layer, and meta information, and persisting it to the unique storage directory path generated in step 2.3.

[0011] Further, in step 2.1, a daemon process is deployed to continuously listen to the perf_event channel; in step 2.2, after the user-space service receives the notification event, it calls the container information acquisition tool docker inspec to obtain the container writable layer and read-only layer paths.

[0012] Further, in step S2, the meta information file is stored at the same level as the writable layer directory and the read-only layer directory when creating the archive structure. When creating the archive structure, it also includes: copying the actual file content of the current writable layer of the container to the writable layer directory, extracting the read-only layer files that coincide with the file paths of the writable layer by traversing each layer of the read-only layer, and saving them as backup copies in the read-only layer for future restoration or analysis; generating a file comparison list and summary of the writable layer and the read-only layer for training and tracing analysis.

[0013] Further, after step S2, the backup content in the generated unique storage directory path is loaded into the running container for resuming the file system to continue training after the training is interrupted by an anomaly.

[0014] Further, the container feature information includes an image, a configuration, a save time, and a training path.

[0015] A computer device includes a processor and a memory, the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0016] A computer readable storage medium storing a computer program, the computer program is executed by a processor to implement the above method.

[0017] Compared with the prior art, the beneficial effects of the present application are: 1、The present application realizes real-time monitoring of file creation behavior in the container by introducing eBPF technology, trains behavior-driven writable layer data backup strategy triggering, and immediately starts the container layer saving process when a certain number of creations are reached, which can effectively avoid the loss of key data or intermediate process data at key nodes, and improve the evaluation and recovery ability of AI model training.

[0018] 2、The AI training container writable layer training data saving method provided by the application can provide an event-driven, automatic perception and on-demand triggering training state archiving mechanism for an AI training process by fusing an eBPF kernel observation mechanism and a container file system snapshot mechanism, triggers training state backup in a "behavior-driven rather than time-driven" manner by monitoring file creation system calls in the training container in real time, can avoid missing key nodes in training by a traditional timing mechanism, and improves data retention capability. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is an implementation flow schematic diagram of the AI training container writable layer training data saving method of the embodiment.

[0020] Figure 2 is a complete implementation flow schematic diagram of the AI training container writable layer training data saving method of the embodiment. DETAILED DESCRIPTION

[0021] The application will be further described below in combination with the accompanying drawings and specific preferred embodiments, but the protection scope of the application is not limited by this.

[0022] For the problem that key training data (located in the container writable layer) in the AI training container process cannot be intelligently saved according to behavior characteristics, the traditional method needs to rely on timing saving or training termination events, and does not have the perception ability of the training behavior process. The application introduces the eBPF technology to realize real-time monitoring of file creation behavior in the container, triggers the writable layer data backup strategy according to the training behavior, and immediately starts the container layer saving process when a certain number of creations is reached, thereby avoiding the loss of key data or intermediate process data of key nodes, and improving the data retention capability, evaluation and recovery capability of the AI model training.

[0023] As shown in Figure 1 , the steps of the AI training container writable layer training data saving method of the embodiment include: Step 1: In the AI model training process, continuously perceive the training behavior in the container, judge whether the saving trigger condition is met, and if the saving trigger condition is met, send a notification event to the user mode service.

[0024] In the embodiment, step 1 can adopt the following steps: Step 1.1: Deploy an eBPF program on the host machine, and mount it to a kernel function related to the file system, including monitoring file creation operation vfs_create or monitoring file opening operation do_sys_open and other kernel hook points.

[0025] The eBPF program is a lightweight sandbox program that runs in the Linux kernel. It can dynamically monitor, filter, or control kernel behavior without modifying the kernel source code or restarting the kernel.

[0026] This embodiment loads the eBPF program into the container host kernel and mounts it to kernel hook points related to the file system, such as `vfs_create` and `do_sys_openat`. When a process within the target container performs a file creation operation, the kernel automatically triggers the eBPF probe function. This allows for real-time capture of file operation behavior generated by the training task in an event-driven manner without modifying the kernel source code. Compared to the traditional polling method based on the ` / proc` file system, this embodiment significantly reduces system overhead and improves real-time detection performance.

[0027] Step 1.2: Use the eBPF program to filter system call behaviors within the target container by identifying the container environment.

[0028] At the host level, multiple containers may run simultaneously. It is necessary to distinguish and accurately capture the system call behavior of the target container to avoid interfering with other containers or host processes. In this embodiment, cgroups and mount namespaces are used to identify the container environment and filter the system call behavior within the target container.

[0029] Specifically, the target cgroup_id (control group ID) and mountnamespace (mount namespace) set by the user-space Python program are first written into the eBPF map. Then, the kernel-space eBPF program reads the user-defined target cgroup_id and mount namespace through lookup, and then obtains the actual cgroup_id and mount namespace of the process that triggered the system call. The two are then compared to see if they match. If they match, file creation is counted; otherwise, it is ignored, ensuring that only the behavior within the target container is counted.

[0030] Step 1.3: The eBPF program maintains a map structure in the kernel. The eBPF program is triggered whenever a new file is created in the container, and the eBPF program records the number of file creations.

[0031] Specifically, the created files include model checkpoints, log files, intermediate cache / temporary files, etc.

[0032] Step 1.4: If the file creation number reaches a preset threshold (e.g., 100), a user space event is triggered, and a notification event is sent to the user space through perf_event, which carries the cgroup_id or container identification information.

[0033] Inside the AI training container, the training process usually generates a large amount of intermediate data and state files, typical scenarios include: 1. Model checkpoint saving, deep learning frameworks (such as TensorFlow, PyTorch) will automatically save model parameters to disk when training to a certain round, this process will correspond to multiple file creation or overwrite operations; 2. Log file writing, the training process will continuously output training curves, loss, accuracy, etc. to the log file or TensorBoard event file, these operations will also trigger file creation or extended writing; 3. Intermediate cache / temporary file generation, during data loading, batch processing, image enhancement, etc. Training frameworks will constantly generate temporary files to support the training process. As can be seen, container training behavior is highly related to file system calls, whether it is checkpoint saving, log flushing or temporary cache generation, it will trigger underlying system calls (such as vfs_create, do_sys_openat, etc.), and the activity level of the training process is strongly related to the frequency of file creation, for example, at the beginning of the training or in the iteration convergence stage, the number of file creations increases significantly; while in the training stagnation or container idle state, the number of file creations will decrease significantly, and the key training stage is often accompanied by a burst of file writing, such as when the model reaches the convergence point, it will trigger large-scale checkpoint writing, therefore, by monitoring file creation behavior, the key nodes of training behavior can be effectively reflected indirectly.

[0034] The embodiment adopts eBPF programs to perceive file creation system calls (such as vfs_create, do_sys_openat, etc.) inside the container in real time, which is equivalent to indirectly capturing internal state changes in the training at the host level: when the number of file creations continues to rise and reaches a threshold (such as 10 new files), it can be determined that the training is in progress and enters a critical stage, therefore, at this time, a user state backup event is triggered, which can ensure snapshot backup at the node where the training produces a large amount of key data, and the above perception method does not need to inject additional monitoring logic inside the container, which can avoid modifying the training framework, while having low overhead and universality.

[0035] Specifically, on the host machine, an eBPF program can be written using BPF Compiler Collection (BCC) or libbpf and mounted to the kernel function vfs_create or do_sys_openat. The program logic is: whenever a new file is created in the container, the eBPF program is triggered; the eBPF program only counts the system call behavior occurring in the specified container through bpf_get_current_cgroup_id() and namespace checks, avoiding affecting other processes, where bpf_get_current_cgroup_id() is used to identify the cgroup ID of the container, and the namespace check is used to verify the isolation environment. The file creation count of each container is saved by the global shared hash table BPF_HASH or single CPU hash table BPF_PERCPU_HASH map in the kernel. When the file creation count reaches a set threshold (such as 100 times), a user space event is triggered. Once the count reaches the threshold, the eBPF program calls the bpf_perf_event_output() function to deliver a notification event to the user space, carrying the cgroup_id or container identification information.

[0036] Step 2: When the user space service receives the notification event, save the writable layer file to obtain the container writable layer archive. The container feature information is extracted and written into the meta information file, and the current container writable layer file and the read-only layer file it covers are saved to the local or remote persistent storage to build a training archive directory structure and generate metadata.

[0037] In this embodiment, step 2 can use the following steps: Step 2.1: The user space service (such as a Python script) listens to the perf_event event; Step 2.2: After receiving the notification, the user space service immediately calls the container runtime tool (such as dockerinspect) to obtain the container writable layer (UpperDir) and read-only layer (LowerDir) path; Step 2.3: Generate a unique storage directory path according to the container ID, timestamp, etc. Step 2.4: Create an archive structure containing the writable layer (UpperDir), read-only layer (LowerDir), and meta information (meta.json), and persist it to the unique storage directory path generated in step 2.3.

[0038] Specifically, the container feature information includes image, configuration, save time, and training path, etc., and the container feature information is extracted and written into the meta information file. That is, the meta information meta.json includes container ID, image name, trigger reason, timestamp, UpperDir layer path, LowerDir layer path, and other container related information. When creating the archive structure, the writable layer (UpperDir) directory and the read-only layer (LowerDir) directory are created, the meta information (meta.json) file is stored at the same level as the writable layer (UpperDir) directory and the read-only layer (LowerDir) directory, the actual file content of the current writable layer (UpperDir) of the container is completely copied into the writable layer (UpperDir) directory, the read-only layer (LowerDir) is traversed, and the read-only layer file with the same file path as the writable layer (UpperDir) directory is copied into the read-only layer (LowerDir) directory, which is used for future restoration or analysis. In the generated unique storage directory, the three parts of the above UpperDir directory, LowerDir directory, and meta.json file constitute the archive structure.

[0039] Further optionally, the file comparison list and digest of the writable layer (UpperDir) and LowerDir (read-only layer) generated by find and sha256sum functions, etc. can also be used for post-training traceability analysis.

[0040] Specifically, in step 2.1, a daemon process (for example, Python + BCC script) can be deployed to continuously monitor the perf_event channel to listen to perf_event events; after the user state service receives the notification, the container information acquisition tool docker inspect is called to obtain the writable layer (UpperDirr) and read-only layer (LowerDir) path of the container; and then in step 2.3, a directory is generated according to the container ID and the current timestamp, for example, / var / backups / container_ <id> _ <timestamp> / to save the current archive data.

[0041] For example, the writable layer (UpperDir) and the read-only layer (LowerDir) paths of the container are obtained by executing docker inspect, and the corresponding contents are copied into the UpperDir and LowerDir subdirectories under the archive directory respectively by using the cp command. At the same time, a meta.json file (meta information file) is automatically generated to record the container ID, image name, trigger reason, timestamp, UpperDir layer path, LowerDir layer path, and other key information. Each key information can be recorded in the form of key-value pair. The following structure can be created in the storage directory: / var / backups / container_ <id> _ <timestamp> / ├── UpperDir / # Writable layer content ├── LowerDir / # Content of the overridden read-only layer └── meta.json# Meta-information Extract container metadata and write it to meta.json, including but not limited to: { "container_id": "abcdef123"; / / Container ID "image": "my-ai-image:v1.2"; / / Image name "trigger_reason": "file_create>= 100"; Trigger reason "timestamp": "2025-06-24T16:33:00"; / / timestamp "upperdir": " / var / lib / docker / overlay2 / ... / diff"; / / UpperDir layer path "lowerdir": " / var / lib / docker / overlay2 / ...: / ... / diff"; / / LowerDir layer path } Optionally, the system can also perform differential analysis and evaluation of training behavior, compare changes in file content in the writable layer (UpperDirr) and the read-only layer (LowerDir), analyze training data generation patterns, behavior frequency, storage bottlenecks, etc., and trace the model training behavior.

[0042] Optionally, the generated unique storage directory backup can also be loaded into the running container to restore the file system and continue training after an abnormal training interruption, enabling rapid recovery of training after an abnormal training interruption.

[0043] like Figure 2 As shown, in a specific application embodiment, when the AI model training container writable layer training data saving is implemented by using the above method of the application, first, the container process runs the training task to generate a file, when the system call triggers vfs_create, do_sys_openat, the kernel BPF level obtains the cgroup_id and mount namespace of the current container, judges whether it matches the target container, if yes, updates the file creation count file_create_count, when the file creation count file_create_count reaches the preset threshold, reports the event to the user state through perf_event; when the user state receives the notification event, the container runtime tool is called to obtain the writable layer (UpperDir) and the read-only layer (LowerDir) path, a unique storage directory is generated according to the container ID and the time, an archive structure containing the writable layer (UpperDir), the read-only layer (LowerDir) and the meta information (meta.json) is created, the current writable layer (UpperDir) file is copied to the writable layer (UpperDir) directory, then the read-only layer (LowerDir) is traversed, the read-only layer file with the same file path as the writable layer (UpperDir) directory is copied to the read-only layer (LowerDir) directory, the container related information such as image, configuration, saving time and training path is recorded to generate meta information (meta.json), and the archive structure is saved in the local or remote storage space to form the final storage structure containing the writable layer (UpperDir), the read-only layer (LowerDir) and the meta information (meta.json).

[0044] The application is aimed at AI training container writable layer training data saving, and provides an event-driven, automatically perceived and on-demand triggered training state archiving mechanism for the AI training process by fusing the eBPF kernel observation mechanism and the container file system snapshot mechanism, can trigger the training state backup in a "behavior-driven rather than time-driven" manner by real-time monitoring of the file creation system call (such as model saving and data generation) in the training container, can avoid missing the key nodes of training by the traditional timing mechanism, thereby effectively avoiding the loss of key data or intermediate process data, and improving the data retention ability, evaluation and recovery ability of the AI model training.

[0045] The embodiment further provides a computer device, including a processor and a memory, the memory is used for storing a computer program, and the processor is used for executing the computer program to execute the above method.

[0046] It can be understood that the above method of the embodiment can be executed by a single device, such as a computer or a server, or can be applied to a distributed scenario and completed by multiple devices in cooperation. In the distributed scenario, one of the multiple devices can only execute one or more steps in the above method of the embodiment, and the multiple devices interact to complete the above method. The processor can be implemented in a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is configured to execute a related program to implement the above method of the embodiment. The memory can be implemented in a read-only memory (ROM), a random access memory (RAM), a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of the embodiment is implemented by software or firmware, the related program code is stored in the memory and executed by the processor.

[0047] The embodiment further provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the above method.

[0048] Those skilled in the art should understand that the above-mentioned embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can be in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer usable program code. The present application is described with reference to flowcharts and / or block diagrams of the methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions that are executed by the processor of the computer or other programmable data processing apparatus generate an apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus with a function specified in one or more flows and / or blocks. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction apparatus that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0049] The above merely preferred embodiments of the present application and not intended to limit the present application in any form. Although the present application has been disclosed with the preferred embodiments as above, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification of the above embodiments made without departing from the technical solution of the present application, according to the technical essence of the present application, should fall within the scope of protection of the technical solution of the present application.< / timestamp> < / id> < / timestamp> < / id>

Claims

1. A method for storing training data in a writable layer of an AI training container, characterized by the following steps: include: Step 1: During the AI ​​model training process, continuously sense the training behavior inside the container, determine whether the save trigger condition is met, and if so, send a notification event to the user-space service. Step 2: When the user-space service receives the notification event, it saves the writable layer file to obtain the container writable layer archive. This is achieved by creating an archive structure containing the writable layer, read-only layer, and metadata. After extracting container feature information and writing it into the metadata file, the current container writable layer file and the overlaid read-only layer file are saved to local or remote persistent storage.

2. The method for saving training data in a writable layer of an AI training container according to claim 1, characterized in that, Step 1 includes: Step 1.1: Deploy the eBPF program on the host machine and mount it to the kernel functions related to the file system. The kernel functions include monitoring the file creation operation vfs_create and monitoring the file opening operation do_sys_open. Step 1.2: The eBPF program identifies the container environment and filters system call behaviors within the target container; Step 1.3: The eBPF program maintains a map structure in the kernel. The eBPF program is triggered whenever a new file is created in the container. The eBPF program records the number of times files are created in the container. Step 1.4: If the number of files created reaches a preset threshold, a user space event is triggered, and a notification event is sent to the user space via perf_event. The notification event carries cgroup_id or container identification information.

3. The method for storing training data in a writable layer of an AI training container according to claim 1, characterized in that, In step S1.2, the eBPF program identifies the container environment by checking bpf_get_current_cgroup_id() and namespace, and only counts system call behavior that occurs in the specified container. bpf_get_current_cgroup_id() is used to identify the cgroup ID of the container, and the namespace check is used to verify the isolation environment. In step S1.3, the number of file creations for each container is stored in the global shared hash table BPF_HASH or the single CPU hash table BPF_PERCPU_HASH map in the kernel.

4. The method for storing training data in a writable layer of an AI training container according to claim 1, characterized in that, Step 2 involves creating an archive structure that includes a writable layer, a read-only layer, and metadata, including: Step 2.1: The user-space service listens for the perf_event notification event; Step 2.2: After receiving the perf_event notification, the user-space service calls the container runtime tool to obtain the paths of the container's writable and read-only layers; Step 2.3: Generate a unique storage directory path based on the container ID and timestamp; Step 2.4: Create an archive structure containing a writable layer, a read-only layer, and metadata, and persist it to the unique storage directory path generated in Step 2.

3.

5. The method for saving training data in a writable layer of an AI training container according to claim 4, characterized in that, In step 2.1, a daemon process is deployed to continuously listen to the perf_event channel; in step 2.2, after receiving the notification event, the user-space service calls the container information acquisition tool docker inspec to obtain the paths of the container's writable and read-only layers.

6. The method for storing training data in a writable layer of an AI training container according to any one of claims 1 to 5, characterized in that, In step S2, when creating the archive structure, the metadata file is stored at the same level as the writable layer directory and the read-only layer directory. The creation of the archive structure also includes: copying the actual file content of the current writable layer of the container to the writable layer directory; extracting the read-only layer files that overlap with the file paths of the writable layer by traversing each layer of the read-only layer and storing them as backup copies in the read-only layer for future restoration or analysis; and generating a file comparison list and summary of the writable layer and the read-only layer for post-training source tracing analysis.

7. The method for storing training data in a writable layer of an AI training container according to any one of claims 1 to 5, characterized in that, Step S2 is followed by loading the backup contents from the generated unique storage directory path into the running container to restore the file system and continue training after an abnormal training interruption.

8. The method for storing training data in a writable layer of an AI training container according to any one of claims 1 to 5, characterized in that, The container feature information includes image, configuration, storage time, and training path.

9. A computer device comprising a processor and a memory, the memory being used to store a computer program, characterized in that, The processor is used to execute the computer program to perform the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 8.

Images