File management method and device

Abstract

The invention provides a file management method and device. The method comprises the following steps: determining a to-be-executed model related task; determining a target mirror image to be referenced according to the model related task; the target mirror image comprises a weight file of a model involved in the model related task; and under the condition that the target mirror image is stored locally, quoting the target mirror image to execute the model-related task. Therefore, repeated downloading and storage of the large-volume weight file during each startup are avoided, and the network bandwidth and the overhead of storage resources are remarkably reduced; meanwhile, through multiplexing of the local mirror image, the environment preparation time is greatly shortened, and the task starting speed is increased.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a file management method and apparatus. Background Technology

[0002] With the widespread application of artificial intelligence technology, large models play a core role in fields such as natural language processing, computer vision, and speech recognition. In real-world business scenarios, model training, fine-tuning, deployment, and iteration occur frequently, with each task relying on a specific version of the model weight file. These weight files are parameter sets formed after the model has been trained on a large amount of data, carrying the model's core capabilities and representing important digital assets for enterprises.

[0003] However, large model weight files are typically very large, and their storage, transmission, and retrieval processes consume significant system resources. Without effective management, this can easily lead to problems such as duplicate file storage, version inconsistencies, slow loading, and delayed task startup. If multiple teams or tasks cannot share and reuse existing weights, it will further waste resources and increase operational costs. Therefore, building an efficient model weight file management mechanism is crucial. Summary of the Invention

[0004] This disclosure provides a file management method and apparatus to at least partially solve one of the technical problems in the related art. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a file management method is provided, the method comprising: determining a model-related task to be executed; determining a target image to be referenced based on the model-related task; the target image including a weight file of a model involved in the model-related task; and, if the target image is stored locally, referencing the target image to execute the model-related task.

[0005] According to a second aspect of the present disclosure, a file management apparatus is provided, the apparatus comprising: a first determining module, configured to determine a model-related task to be executed; a second determining module, configured to determine a target image to be referenced based on the model-related task; the target image including a weight file of a model involved in the model-related task; and a processing module, configured to reference the target image to execute the model-related task if the target image is stored locally.

[0006] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the file management method as described in the first aspect of the present disclosure.

[0007] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a file management method as described in the first aspect of the present disclosure.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the file management method as described in the first aspect of the present disclosure.

[0009] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects: In this technical solution, the target image to be referenced is determined based on the model-related tasks to be executed. The target image contains the weight files of the models on which the model-related tasks depend, achieving precise location of the required model weight files. If the target image already exists locally, it is directly referenced to execute the task, avoiding repeated downloading and storage of large weight files each time the task starts, significantly reducing network bandwidth and storage resource overhead. Simultaneously, by reusing local images, environment preparation time is greatly shortened, and task startup speed is accelerated. The target image is built based on a base image and at least one image layer, where the image layer contains basic weight update files, enabling customized extensions of the base image and allowing the target image to be built on demand. This enhances the flexibility and adaptability of image generation. The completed image is saved locally, providing a directly callable runtime environment for subsequent identical or similar tasks, further supporting efficient reuse and rapid deployment. Furthermore, based on the referenced target image and model-related tasks, when building a task image, an image layer is overlaid on the target image to obtain the task image. This image layer includes the task processing logic and runtime environment for model-related tasks, achieving decoupling of basic resources and business logic and improving system modularity. The task image inherits the model weights and basic environment from the target image, ensuring consistency and reproducibility in execution. Simultaneously, different tasks only require building different image layers, avoiding the repeated storage of large weight files and saving storage space and build time.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0012] Figure 1This is a flowchart illustrating the file management method shown in the first embodiment of this disclosure; Figure 2 This is a flowchart illustrating the file management method shown in the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the file management method shown in the third embodiment of this disclosure; Figure 4 This is a flowchart illustrating the file management method shown in the fourth embodiment of this disclosure; Figure 5 This is a flowchart illustrating the file management method shown in the fifth embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of the base image corresponding to the different models shown in the embodiments of this disclosure; Figure 7 This is a schematic diagram of the structure of the task image shown in the embodiments of this disclosure; Figure 8 This is a schematic diagram of the mirrored retrieval of model-related tasks shown in the embodiments of this disclosure; Figure 9 This is a schematic diagram of the structure of the file management device shown in the sixth embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0013] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0014] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0015] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0016] Because large model weight files are typically enormous, ranging from hundreds of GB to even TB, their storage, transmission, and retrieval processes consume significant system resources. In the early stages of large model development, companies typically ran a single large model, sufficient for their needs. In this context, the management of the basic weight files was often simple: storing the basic weight files directly on a server and retrieving them as needed, or packaging them together with the runtime code for direct use. However, as major companies released open-source large models and promoted their adoption, companies began deploying multiple large models. Furthermore, their usage expanded beyond simply meeting the original capabilities of open-source models; they increasingly used their own internal data for fine-tuning, ensuring the large models performed well within their specific business domains. In this context, companies generally opted to centrally manage the basic weight files in a file repository (e.g., Minio / S3), retrieving them locally for use, or mounting them directly using a file-sharing tool like a Network File System (NFS).

[0017] However, the unified management via file repository may result in significant local redundancy. When performing multiple fine-tuning tasks or deploying multiple large model services, different tasks may pull the same model's base weight file, leading to the same model's base weight file being stored in multiple locations on the same machine. In cases where the base weight file of a large model is huge, this will cause serious resource redundancy, and the base weight file may be scattered in various places, making it difficult to clean up. Mounting and directly reading via NFS may have issues such as the inability to pull in advance and the inability to reuse the data. When multiple fine-tuning tasks use the same model on the same machine, or multiple services use the same model and are deployed on the same machine, each task needs to rely on bandwidth to read from NFS in real time. Since the model's base weight file is huge, such frequent reading will seriously consume network bandwidth, and at the same time, such frequent reading has very high bandwidth requirements.

[0018] To address at least one of the aforementioned problems, this disclosure provides a file management method and apparatus. The file management method and apparatus according to embodiments of this disclosure are described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating the file management method shown in the first embodiment of this disclosure.

[0020] like Figure 1 As shown, the file management method includes the following steps: Step 101: Determine the model-related tasks to be executed.

[0021] To improve the accuracy of task execution, in this embodiment of the disclosure, the model-related tasks that need to be executed are obtained. For example, users can submit model-related tasks through the platform interface, or trigger model-related tasks through the application programming interface (API). Model-related tasks may include at least one of the following: model fine-tuning tasks, model deployment tasks, etc.

[0022] Among them, model fine-tuning tasks refer to the process of further training a pre-trained model using data from a specific task or domain to better adapt it to a specific application scenario. Model fine-tuning tasks may include, but are not limited to, text generation fine-tuning tasks, image generation fine-tuning tasks, and speech fine-tuning tasks. Model deployment tasks refer to the process of integrating the trained or fine-tuned model into the production environment so that it can receive input and return inference results to support actual business applications. Model deployment tasks may include, but are not limited to, image classification model deployment and speech recognition model deployment.

[0023] Step 102: Determine the target image to be referenced based on the model-related tasks; the target image includes the weight file of the model involved in the model-related tasks.

[0024] To achieve precise matching between task and model weight files, ensuring the correct model version is used and avoiding performance deviations due to version errors, one possible approach is to determine the model type and version associated with the task based on its configuration file. Then, based on this model type and version, a standardized mirror image containing the weight files of all models involved in the task—the target image—is located and determined, serving as the foundation for the execution environment. Specifically, the weight files of the models involved in the task must match both the model type and version.

[0025] It should be noted that, in order to improve the accuracy of task execution, the number of models involved in model-related tasks is at least one; the number of target images is at least one, and at least one model corresponds one-to-one with at least one target image.

[0026] In other words, model-related tasks may involve one or more models (i.e., at least one). For example, a task may require the simultaneous use of a language understanding model and a generative model to complete the task. Accordingly, there is at least one target image in the system. These models and target images maintain a one-to-one correspondence, meaning that each model is carried and managed by a unique target image. This ensures accurate matching of model version, configuration, and dependent environment, avoids confusion or mismatch between models and images, and guarantees the determinism and traceability of task execution.

[0027] To improve the flexibility of the target image, in this embodiment of the disclosure, the target image is a base image; or, the target image is constructed based on the base image and at least one image layer, wherein the base image includes the model's base weight file and the image operating system, and the at least one image layer includes a base weight update file.

[0028] It should be noted that the base weight file includes the original parameters learned by the pre-trained model on a large-scale general dataset; the base weight update file includes the parameter differences for incremental adjustment of the base weights. It usually does not contain the complete model parameters, but only records the parameter changes generated after training on specific task or domain data. It is used to merge with the base weights in the base weight file during inference, thereby achieving efficient and lightweight model customization and updates.

[0029] As an example, the target image can be a base image that directly contains the model's base weights file and the operating system, used to perform model-related tasks without additional adjustments.

[0030] As another example, a customized target image can be constructed by overlaying one or more image layers on top of a base image. Each image layer can contain a base weight update file, which can not only be used to correct or enhance the original parameters in the base weights, but also contain incremental parameters generated after the model is trained on new data or a new task. Thus, the target image can complete inference at runtime by loading the original weights of the base image and fusing the incremental parameters, realizing the direct reuse of the base image, while supporting efficient and flexible model customization and version management without modifying the original model weights.

[0031] Step 103: If the target image is stored locally, reference the target image to perform model-related tasks.

[0032] To avoid repeatedly transmitting and storing the model's base weight file, in this embodiment, when the target image is stored locally, the container engine can directly reference the target image to execute model-related tasks. Since the target image is a base image; or, the target image is built from the base image and at least one image layer—meaning the target image fully encapsulates the model's base weight file from the base image, or encapsulates the model's base weight file from the base image along with customized incremental update parameters—there is no need to re-download the model's base weight file or perform additional fine-tuning operations, significantly reducing network bandwidth and storage resource overhead. Simultaneously, by reusing the locally stored target image, environment preparation time is greatly shortened, and task startup speed is accelerated.

[0033] As an example, such as Figure 2 As shown, step 103 may include the following steps: Step 1031: If the target image is stored locally, reference the target image.

[0034] To avoid repeatedly downloading the model's base weight file, in this embodiment of the disclosure, if the target image is stored locally, the target image is directly referenced.

[0035] It should be noted that if the target image is not stored locally, the target image is pulled from the image repository; the target image is referenced to perform model-related tasks; and the target image is stored locally.

[0036] In other words, to achieve on-demand retrieval and rapid deployment, if the target image is not stored locally, the required target image can be pulled from the image repository. After the pull is complete, the target image can be directly referenced to start a container instance and execute model-related tasks. Furthermore, after the first pull or after each successful task execution, the target image will be automatically stored in the local image repository for subsequent reuse.

[0037] Step 1032: Construct a task image based on the referenced target image and the model-related tasks.

[0038] To enable the reuse of the model's base weight file, one possible approach is to further customize and generate a task image specifically for the current model's related tasks, based on the locally stored target image.

[0039] In summary, when the target image is stored locally, the target image can be directly referenced and task images can be built based on the model-related tasks. This eliminates the need to repeatedly download or transfer large basic weight files, saving network bandwidth and loading time. It also avoids the repeated packaging of the basic weight files for the entire model, enabling the rapid generation of dedicated task images for different application scenarios. This improves the reusability of basic weight files while maintaining the independence and consistency of each task environment, making management and maintenance easier.

[0040] In some embodiments, such as Figure 3 As shown, step 1032 may include the following steps: Step 10321: Obtain the task processing logic and runtime environment of the model-related tasks.

[0041] To ensure the stability, consistency, and reproducibility of task execution and avoid operational failures due to missing dependencies or configuration errors, this embodiment of the disclosure defines and prepares the task processing logic and runtime environment required to complete the task before executing model-related tasks. The task processing logic includes data input parsing, preprocessing methods, model inference invocation methods, output post-processing rules, and business judgment logic; while the runtime environment may include the required programming language version, third-party libraries and their versions, configuration parameters, and environment variables.

[0042] Step 10322: Overlay an image layer onto the target image to obtain the task image; the image layer includes the task processing logic and the runtime environment.

[0043] To decouple model capabilities from task logic and improve the reusability of the model's base weight files, one possible approach during model deployment and service-oriented architecture is to build a task image by overlaying a new image layer on top of the target image. The target image already encapsulates the model's base weight files and possesses the model's core capabilities; while the newly added image layer integrates the task processing logic and runtime environment required for the current task.

[0044] Step 10323: Run the task image to perform model-related tasks.

[0045] To achieve standardized, efficient, and scalable deployment of model services, container instances of task images are launched through a container engine to execute model-related tasks. These task images are built on a layered architecture, fully encapsulating the underlying target image, task-specific processing logic, and the required runtime environment. When the container instance starts, it automatically loads the model, initializes the service process, executes model-related tasks according to preset logic, and returns execution results in real time.

[0046] In summary, by acquiring the task processing logic and runtime environment of model-related tasks, and overlaying an image layer containing the task processing logic and runtime environment onto the target image, a task image is formed and run to execute the task. This decouples model capabilities from task logic, allowing the same target image to be reused for multiple different tasks, thus improving model reusability. Building task images through a layered overlay method only requires adding lightweight incremental content, reducing redundant data transmission and storage overhead, and accelerating image construction and deployment. The task-related processing logic and runtime environment are fully encapsulated within the image, ensuring consistency and portability of the execution environment and avoiding dependency conflicts. Finally, by running the task image to execute model-related tasks, the efficiency, flexibility, and reliability of model-related task execution are significantly improved.

[0047] In some embodiments, such as Figure 4 As shown, the following steps may be included after step 103: Step 104: Detect whether the target image or image layer in local storage meets the set clearing conditions; wherein, the clearing conditions include at least one of the following: it has not been referenced by model-related tasks within a continuous time period exceeding the first duration threshold; all model-related tasks referencing the target image or image layer have been completed.

[0048] To automatically identify and release storage space to avoid resource waste, one possible approach is to detect the target image or image layer in local storage and determine whether it meets the set cleanup conditions to determine whether the target image or image layer can be cleaned up.

[0049] The clearing conditions include, but are not limited to: (1) If the target image or image layer has not been referenced by any model-related task for a continuous period of time exceeding the first duration threshold, such as not being pulled or started for 7 or 30 consecutive days, it indicates that it is currently in an idle state and is a resource that can be cleared. (2) All model-related tasks that reference the target image or image layer have been completed and are in a terminated state, that is, there are no active tasks that are running, waiting to be scheduled or associated, indicating that their service lifecycle has ended.

[0050] Step 105: If the target image or image layer meets the clearing conditions, perform the clearing process.

[0051] To free up storage space and improve resource utilization efficiency, as a possible approach, the target image or image layer in local storage can be automatically cleaned up if it meets preset cleanup conditions.

[0052] In summary, by setting clear cleanup conditions, target images or image layers that are no longer needed in local storage can be cleaned up in a timely manner, effectively freeing up disk space and avoiding the waste of storage resources. The cleanup condition is met when an image has not been referenced for a continuous period of time exceeding a first time threshold, or when all its associated model tasks have been completed. This cleanup condition takes into account both the time dimension and the task status dimension, which not only prevents long-term unused images from occupying resources, but also ensures that running tasks are not affected, thus improving the security and rationality of resource reclamation.

[0053] To avoid accidental deletion of the target image, in some embodiments, it is necessary to determine whether the target image meets the deletion conditions, such as... Figure 5 As shown, before step 104, the following steps are also included: Step 106: For the target image stored locally, determine the duration of the continuous time interval between the historical time point when the target image is referenced by model-related tasks and the current time point, and determine whether all model-related tasks referencing the target image have been completed.

[0054] To accurately identify resource usage status, one possible approach is to analyze the historical records of the target image being referenced by model-related tasks in local storage to determine the specific time point of the last reference, i.e., the historical time point, and calculate the length of the continuous unreferenced time period (i.e., idle time) between the historical time point and the current time point. At the same time, it is detected whether there are currently any tasks that are still running, waiting, or scheduled that depend on the target image, in order to determine whether all model-related tasks referencing the target image have been completed.

[0055] Step 107: If the duration exceeds the first duration threshold, or if all model-related tasks referencing the target image have been completed, determine that the target image meets the clearing conditions.

[0056] To ensure timely resource release, the target image in local storage can be determined to meet the cleanup criteria if any of the following conditions are met: (1) If the length of the continuous unreferenced time period (i.e., idle time) between the historical time point and the current time point exceeds the first time threshold, the target image is determined to meet the clearing condition; (2) If all model-related tasks referencing the target image have been completed, determine that the target image meets the cleanup conditions.

[0057] Step 108: If the duration does not exceed the first duration threshold and the model-related tasks referencing the target image have not been completed, determine that the target image does not meet the clearing conditions.

[0058] In order to achieve secure management and reasonable reclamation of local storage resources, if the length of the continuous unreferenced time period (i.e., idle time) between the historical time point and the current time point does not exceed the first time length threshold, and the model-related tasks referencing the target image have not been completed, it is determined that the target image does not meet the clearing conditions.

[0059] In summary, by comprehensively judging the continuous unreferenced duration of the target image and the execution status of model-related tasks, the system determines whether the target image in local storage meets the cleanup conditions, thus achieving fine-grained management of local storage resources. When the duration exceeds a first threshold or all referenced tasks are completed, the target image is determined to meet the cleanup conditions and can be cleaned, thereby timely releasing long-term idle or de-used image resources and avoiding storage waste. Conversely, if the duration does not exceed the threshold but some tasks are still incomplete, the image is deemed ineligible for cleanup, effectively ensuring the integrity and stability of running services and preventing service interruptions caused by accidental cleanup. This mechanism balances resource reclamation efficiency and system security, improving the automation and fine-grained level of model image lifecycle management. Based on any of the above embodiments, the file management method of this disclosure may include the following steps: (1) Building the base image like Figure 6 As shown, by using a mirroring method, the basic weight files of each large model are saved as a base image, which can be referenced by subsequent model fine-tuning or deployment tasks. A fine-tuning or deployment environment is then built upon this image to perform the model fine-tuning or deployment tasks. Each image contains only the basic image operating system and the weight files of the large model; it does not contain any code or code execution environment. (2) Build a fine-tuning or deployment task image like Figure 7 As shown, on top of the base image containing the model's base weight file, corresponding fine-tuning code and the runtime environment required for the fine-tuning code can be added. Based on the layering mechanism of the image itself, each build will form an independent layer. The image is assembled from various layers, so the image for fine-tuning or deployment tasks is a reference to the base image in the base weight file section, and will not be saved repeatedly.

[0060] (3) Make fine adjustments or deployments like Figure 8 As shown, when initiating multiple model fine-tuning or deployment tasks, images are pulled from the image repository. Due to the image layering mechanism, each layer is only pulled locally once (when downloading an image from the image repository, the repository checks if a certain layer is already available locally). The corresponding images are reused, and regardless of how many tasks reuse them, they are only pulled once. In contrast, this solution saves the model's basic weights as a layer. That is, for all tasks that use a certain model base, only one copy is pulled and saved locally, and the corresponding tasks are reused. Even if the same task is executed multiple times, or multiple tasks reference the same model, only one copy is saved locally, achieving the effect of "caching" and reuse.

[0061] (4) Cleaning Because images are associated with the containers they are used in, it is possible to identify whether an image is in use. Images that are no longer in use can be cleaned up directly. If an image has corresponding task references, a corresponding prompt will be given. After deleting the corresponding task, the image can be deleted. In addition, the next time the task is executed, if the image is not stored locally, it will be automatically downloaded from the image repository without affecting the execution of the task.

[0062] Corresponding to the file management method provided in the above embodiments, this disclosure also provides a file management device. Since the file management device provided in this disclosure corresponds to the file management method provided in the above embodiments, the implementation of the file management method is also applicable to the file management device provided in this disclosure, and will not be described in detail in this disclosure.

[0063] Figure 9 This is a schematic diagram of the structure of the file management device shown in the sixth embodiment of this disclosure.

[0064] like Figure 9 As shown, the file management device 900 includes: a first determination module 910, a second determination module 920, and a processing module 920.

[0065] The first determining module 910 is used to determine the model-related task to be executed; the second determining module 920 is used to determine the target image to be referenced based on the model-related task; the target image includes the weight file of the model involved in the model-related task; and the processing module 930 is used to reference the target image to execute the model-related task if the target image is stored locally.

[0066] As one possible implementation of this disclosure, the file management device 900 further includes a construction module.

[0067] The construction module is used to determine the base image and at least one image layer in the target image when the target image is not stored locally and does not exist in the image repository; pull the base image and the at least one image layer from the image repository; and construct the target image based on the base image and the at least one image layer.

[0068] As one possible implementation of this disclosure, the file management device 900 further includes a storage module.

[0069] The storage module is used to retrieve the target image from the image repository when the target image is not stored locally; reference the target image to perform the model-related tasks; and store the target image locally.

[0070] As one possible implementation of this disclosure, the target image is a base image; or, the target image is constructed based on the base image and at least one image layer; the base image includes the model's base weight file and the image operating system; at least one image layer includes a base weight update file.

[0071] As one possible implementation of this disclosure, the processing module 930 is used to reference a target image; construct a task image based on the referenced target image and model-related tasks; and run the task image to execute model-related tasks.

[0072] As one possible implementation of this disclosure, the processing module 930 is used to obtain the task processing logic and operating environment of the model-related task; and to overlay a mirror layer on the target mirror to obtain a task mirror; the mirror layer includes the task processing logic and the operating environment.

[0073] As one possible implementation of this disclosure, the model-related tasks include at least one of the following: model fine-tuning tasks and model deployment tasks; the number of models involved in the model-related tasks is at least one; the number of target images is at least one; and at least one model corresponds one-to-one with at least one of the target images.

[0074] As one possible implementation of this disclosure, the file management device 900 further includes a clearing module.

[0075] The clearing module is used to perform clearing processing on a target image or image layer stored locally, provided that the target image or image layer meets the clearing conditions. The clearing conditions include at least one of the following: it has not been referenced by model-related tasks within a continuous time period exceeding a first duration threshold; all model-related tasks referencing the target image or image layer have been completed.

[0076] As one possible implementation of this disclosure, for a target image stored locally, the clearing conditions include: not being referenced by model-related tasks within a continuous time period exceeding a first duration threshold, and all model-related tasks referencing the target image or the image layer being completed; the file management device 900 further includes: a third determining module.

[0077] The third determining module is used to determine, for the target image stored locally, the duration of a continuous time interval between the historical time point and the current time point when the target image is referenced by model-related tasks, and whether all model-related tasks referencing the target image have been completed; if the duration exceeds a first duration threshold, or if all model-related tasks referencing the target image have been completed, the target image is determined to meet the clearing condition; if the duration does not exceed the first duration threshold and the model-related tasks referencing the target image have not been completed, the target image is determined not to meet the clearing condition.

[0078] The file management device of this disclosure determines the target image to be referenced based on the model-related task to be executed. The target image contains the weight files of the models on which the model-related task depends, achieving precise location of the required model weight files. If the target image already exists locally, the task can be executed directly using the image, avoiding repeated downloading and storage of large weight files each time the task starts, significantly reducing network bandwidth and storage resource overhead. Simultaneously, by reusing local images, the environment preparation time is greatly shortened, and the task startup speed is accelerated.

[0079] In an exemplary embodiment, an electronic device is also proposed.

[0080] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the file management method as proposed in any of the foregoing embodiments.

[0081] As an example, Figure 10 This is a schematic diagram of the structure of an electronic device 1000 as shown in an exemplary embodiment of this disclosure, as follows: Figure 10 As shown, the aforementioned electronic device 1000 may further include: The memory 1010 and the processor 1020 are connected by a bus 1030, which connects different components (including the memory 1010 and the processor 1020). The memory 1010 stores a computer program, and when the processor 1020 executes the program, it implements the file management method described in the embodiments of this disclosure.

[0082] Bus 1030 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0083] Electronic device 1000 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 1000, including volatile and non-volatile media, removable and non-removable media.

[0084] The memory 1010 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1040 and / or cache memory 1050. The server may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1060 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1030 via one or more data media interfaces. Memory 1010 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0085] A program / utility 1080 having a set (at least one) of program modules 1070 may be stored in, for example, memory 1010. Such program modules 1070 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1070 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0086] Electronic device 1000 can also communicate with one or more external devices 1090 (e.g., keyboard, pointing device, display 1091, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1092. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1093. As shown, network adapter 1093 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0087] The processor 1020 performs various functional applications and data processing by running programs stored in the memory 1010.

[0088] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the file management method of this disclosure embodiment, and will not be repeated here.

[0089] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to perform the file management method proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0090] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the file management method proposed in any of the above embodiments.

[0091] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0092] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A file management method, characterized in that, The method includes: Identify the model-related tasks to be executed; Based on the model-related tasks, a target image to be referenced is determined; the target image includes the weight file of the model involved in the model-related tasks. If the target image is stored locally, the target image is referenced to perform the model-related tasks.

2. The method according to claim 1, characterized in that, The method further includes: If the target image is not stored locally and does not exist in the image repository, determine the base image and at least one image layer in the target image; Pull the base image and the at least one image layer from the image repository; The target image is constructed based on the base image and the at least one image layer.

3. The method according to claim 1, characterized in that, The method further includes: If the target image is not stored locally, pull the target image from the image repository; The target image is referenced to perform the model-related tasks; The target image is stored locally.

4. The method according to claim 1, characterized in that, The target image is a base image; or, the target image is constructed based on the base image and at least one image layer. The base image includes the model's base weight file and the image operating system; The at least one mirror layer includes a basic weight update file.

5. The method according to claim 1, characterized in that, The step of referencing the target image to perform the model-related tasks includes: Referencing the target image; Based on the referenced target image and the model-related tasks, construct a task image; Run the task image to perform the model-related tasks.

6. The method according to claim 5, characterized in that, The step of constructing a task image based on the referenced target image and the model-related tasks includes: Obtain the task processing logic and runtime environment of the tasks related to the model; An image layer is superimposed on the target image to obtain the task image; the image layer includes the task processing logic and the runtime environment.

7. The method according to claim 1, characterized in that, The model-related tasks include at least one of the following: model fine-tuning task and model deployment task; the number of models involved in the model-related tasks is at least one; the number of target images is at least one; the at least one model corresponds one-to-one with the at least one target image.

8. The method according to claim 1, characterized in that, The method further includes: For a target image or image layer stored locally, a clearing process is performed if the target image or image layer meets the clearing conditions. The clearing conditions include at least one of the following: it has not been referenced by model-related tasks within a continuous time period exceeding a first duration threshold; all model-related tasks referencing the target image or image layer have been completed.

9. The method according to claim 8, characterized in that, For the target image stored locally, the clearing conditions include: not being referenced by model-related tasks within a continuous time period exceeding a first duration threshold, and all model-related tasks referencing the target image or the image layer having been completed; the method further includes: For the target image stored locally, determine the duration of the continuous time interval between the historical time point and the current time point when the target image is referenced by model-related tasks, and determine whether all model-related tasks referencing the target image have been completed. If the duration exceeds the first duration threshold, or if all model-related tasks referencing the target image are completed, the target image is determined to meet the clearing conditions. If the duration does not exceed the first duration threshold and the model-related tasks referencing the target image have not been completed, it is determined that the target image does not meet the clearing conditions.

10. A document management device, characterized in that, The device includes: The first determination module is used to determine the model-related tasks to be executed; The second determining module is used to determine the target image to be referenced based on the model-related task; the target image includes the weight file of the model involved in the model-related task; The processing module is used to reference the target image to perform the model-related tasks when the target image is stored locally.

Images