Model file processing method and electronic device

Abstract

The model file processing method and the electronic equipment provided by the application relate to the technical field of artificial intelligence, and the weight file is distributed according to the read-write bandwidth of multiple storage devices of a server, so that the data amount borne by each storage device matches the read-write bandwidth, and the input-output bottleneck caused by uneven load is avoided. In the file loading stage, cross-device file parallel reading is realized based on storage path information, and the concurrent task depth of the same storage device is controlled, and the loading tasks of different storage devices are allowed to be parallel, the bandwidths of multiple storage devices are aggregated, and the data reading throughput is improved. Then, the weight files read in parallel are combined in the order of the original model structure, so as to ensure the correctness and integrity of model loading. Therefore, the application can utilize the server hardware resources at low cost and high efficiency, shorten the loading time of model service startup, and improve the response speed of model service deployment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for processing model files and an electronic device. Background Technology

[0002] With the development of artificial intelligence technology, Large Language Models (LLMs) based on the Transformer architecture have become a core driving force. However, the continuous growth in the scale of model parameters poses a challenge to the efficiency of loading the model's weight files. Weight file loading is the process of loading weights from persistent storage into processor memory during the service cold start phase; therefore, weight file loading has become a key bottleneck affecting service readiness time. Current weight file loading methods do not fully utilize hardware resources and rely on dedicated storage hardware, making it impossible to achieve fast weight file loading at low cost and high efficiency. Summary of the Invention

[0003] This application provides a method for processing model files and an electronic device to at least solve the problem in related technologies of how to utilize server hardware resources at low cost and high efficiency to achieve fast loading of weight files.

[0004] This application provides a method for processing model files, including: Based on the read / write bandwidth of each of the multiple storage devices in the server, the multiple weight files of the model to be processed are allocated to the multiple storage devices, and the storage path information corresponding to each of the multiple weight files is generated. The total number of weight files allocated to the multiple storage devices is positively correlated with the read / write bandwidth of each of the multiple storage devices. In response to the model loading command, based on the storage path information, multiple weight files are loaded from multiple storage devices to the server memory in parallel. The number of loading tasks executed in parallel on the same storage device is less than the preset concurrent queue depth, and loading tasks are executed in parallel on different storage devices. The multiple weight files loaded into memory are combined in the order of the original model structure corresponding to the model to be processed to obtain the loaded weight file.

[0005] This application also provides a model file processing apparatus, including: The allocation unit is used to allocate multiple weight files of the model to be processed to multiple storage devices according to the read and write bandwidth of each storage device in the server. The generation unit is used to generate storage path information corresponding to each of the multiple weight files. The total number of weight files allocated to the multiple storage devices is positively correlated with the read and write bandwidth of each of the multiple storage devices. The loading unit is used to respond to the model loading instruction and, based on the storage path information, execute in parallel the loading tasks of loading multiple weight files from multiple storage devices to the memory of the server. The number of loading tasks executed in parallel on the same storage device is less than the preset concurrent queue depth, and loading tasks are executed in parallel on different storage devices. The combination unit is used to combine multiple weight files loaded into memory according to the original model structure corresponding to the model to be processed, so as to obtain the loaded weight file.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described model file processing methods.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of any of the above-described model file processing methods.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model file processing methods.

[0009] The model file processing method and electronic device provided in this application distribute weight files according to the read / write bandwidth of multiple storage devices on the server, ensuring that the data volume carried by each storage device matches its read / write bandwidth. This avoids input / output bottlenecks caused by uneven load. During the file loading phase, parallel file reading across devices is achieved based on storage path information. By controlling the concurrent task depth on the same storage device and allowing parallel loading tasks between different storage devices, the bandwidth of multiple storage devices is aggregated, improving data read throughput. Subsequently, the concurrently read weight files are combined in the order of the original model structure to ensure the correctness and integrity of model loading. Therefore, this application can utilize server hardware resources at low cost and high efficiency, shorten the loading time for model service startup, and improve the response speed of model service deployment. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a method for processing model files provided in an embodiment of this application; Figure 2A model file loading process topology diagram provided in this application embodiment; Figure 3 This is a schematic diagram of a model file processing device provided in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] This application provides a method and electronic device for processing model files, aiming to solve the problem of slow model service startup caused by storage input / output (I / O) bottlenecks when loading weight files for artificial intelligence models such as LLM. This application achieves efficient and concurrent loading of weight files by fully utilizing the hardware resources of multiple storage devices configured locally on the server through software-level scheduling.

[0015] Before model deployment, this application intelligently distributes the model's weight file set to multiple local storage devices on the server, such as hard drives. During loading, the loader concurrently reads the weight file fragments allocated to each storage device and aggregates them in memory, thereby converting the serial I / O of a single storage device into parallel I / O of multiple storage devices, greatly improving loading throughput.

[0016] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] Figure 1 This document provides a flowchart illustrating a method for processing model files, as provided in an embodiment of this application. The method is described in detail below, taking into account the execution flow of the model file processing method.

[0018] Step 101: Based on the read / write bandwidth of each of the multiple storage devices in the server, allocate the multiple weight files of the model to be processed to the multiple storage devices, and generate the storage path information corresponding to each of the multiple weight files. The total number of weight files allocated to the multiple storage devices is positively correlated with the read / write bandwidth of each of the multiple storage devices.

[0019] In the embodiments of this application, a server is a physical or virtual device used to deploy and run large model services, and it has multiple storage devices configured internally or locally for data access. A storage device refers to a unit within the server that has persistent data storage capabilities, such as a hard drive. Storage devices are typically mounted as independent directories in the server's operating system. Different storage devices may have different read / write bandwidths due to factors such as interface type, internal flash memory chip quality, or mechanical structure.

[0020] The model to be processed refers to the LLM or other deep learning model that needs to be loaded. Its parameters are huge and consist of multiple independent files, which are called weight files. The weight files store the parameters (i.e. weights) obtained after the model is trained.

[0021] When allocating weighted files, the total amount of data in the weighted files ultimately allocated to a storage device should be positively correlated with the read / write bandwidth of that storage device. That is, storage devices with higher read / write bandwidth will be allocated more or larger weighted files; conversely, storage devices with lower read / write bandwidth will be allocated fewer weighted files.

[0022] After distributing the weight files to various storage devices, it is also necessary to generate storage path information, which records the specific path of the storage device where each weight file is finally stored. For example, a weight file with the original name model-00001-of-00016.safetensors can be recorded as having a storage path of / disk_b / weights / model-00001.safetensors.

[0023] Step 102: In response to the model loading instruction, load tasks are executed in parallel from multiple storage devices to the server memory according to the storage path information. The number of load tasks executed in parallel on the same storage device is less than the preset concurrent queue depth, and load tasks are executed in parallel on different storage devices.

[0024] In the embodiments of this application, the model loading instruction is a command triggered when starting the model service or resuming model training, used to instruct the weight file to be read from the persistent storage device into memory. Memory refers to the server's system memory, which serves as a transfer station for files from storage devices to processor memory, or is used directly for graphics processing unit (CPU) inference scenarios.

[0025] After responding to the model loading instruction, based on the storage path information, all weight files are split into independent loading tasks and these loading tasks are executed concurrently. The storage path information can be used to determine the storage device corresponding to each loading task, thereby implementing concurrency control.

[0026] The preset concurrent queue depth is a pre-defined threshold representing the maximum number of load tasks allowed to execute simultaneously on a single storage device. While storage devices support parallel processing of multiple I / O requests, performance may degrade due to internal resource contention when the number of concurrent requests exceeds the optimal depth of their hardware design. Therefore, by limiting the number of concurrent tasks on the same storage device to below the preset concurrent queue depth, it is ensured that each storage device operates at its best. Furthermore, different storage devices, possessing independent hardware resources and data channels, can execute load tasks simultaneously without conflict, thus achieving parallel execution of load tasks.

[0027] Step 103: Combine the multiple weight files loaded into memory according to the original model structure order corresponding to the model to be processed to obtain the loaded weight file.

[0028] In the embodiments of this application, since all weight files are stored in a distributed manner and loaded concurrently, the weight files may initially be in an unordered state in memory. Therefore, data aggregation is required.

[0029] The original model structure order refers to the inherent arrangement of components (such as embedding layers, multiple Transformer layers, output layers, etc.) in the model being processed during definition. Each weight file contains tensors (such as weight matrices and bias terms) corresponding to a specific part of the model structure. Based on the model's configuration file or metadata, the model position of each weight file's tensors is identified, and then they are reassembled into a complete model state dictionary (state_dict) according to the original model structure order. This ensures that the scattered weight files can be correctly reconstructed into the complete parameter set corresponding to the model. The final loaded weight file is the structurally correct set of weight data.

[0030] This application intelligently distributes weighted files based on the read / write bandwidth of storage devices and enables concurrent loading across devices during loading. This transforms the serial loading limited by the bandwidth of a single device into parallel loading across multiple devices, thereby improving data loading throughput and shortening the startup time of the model service. By pre-setting the concurrent queue depth as a threshold for the same storage device, performance degradation caused by excessive concurrency is avoided, ensuring that each storage device is in optimal condition, thus maximizing hardware resource utilization. Therefore, server hardware resources can be utilized at low cost and high efficiency, reducing the loading time for model service startup.

[0031] In one possible implementation of this application embodiment, when allocating multiple weight files of the model to be processed to multiple storage devices, the following methods can be used, but are not limited to: the ratio of the read / write bandwidth of each of the multiple storage devices to the total bandwidth is processed to obtain the read / write bandwidth ratio of each of the multiple storage devices, wherein the total bandwidth is the sum of the read / write bandwidth of each of the multiple storage devices; based on the read / write bandwidth ratio and the file size of each of the multiple weight files, the multiple weight files are allocated to the multiple storage devices, wherein the higher the read / write bandwidth ratio of the storage device, the more weight files are allocated.

[0032] In the embodiments of this application, read / write bandwidth refers to the maximum data transfer rate that each storage device can achieve under continuous data read / write operations, which can reflect the actual I / O performance of each storage device. The total bandwidth is obtained by summing the read / write bandwidths of all storage devices participating in the weighted file allocation, and it represents the upper limit of the server's aggregate I / O capability under ideal conditions.

[0033] Ratio processing involves dividing the read / write bandwidth of a single storage device by the total bandwidth to obtain the read / write bandwidth ratio of the storage devices. This read / write bandwidth ratio reflects the relative performance weight of each storage device in the server.

[0034] Each weighted file has a specific file size. During the allocation process, the size of each weighted file and the read / write bandwidth ratio of each storage device need to be considered comprehensively. Ultimately, the total number of weighted files stored on each storage device (i.e., the sum of the sizes of all weighted files allocated to that storage device) is proportional to its read / write bandwidth ratio.

[0035] Specifically, for example, all weighted files can be sorted by size first, and then a weighted round-robin approach can be used to allocate them sequentially, starting with the largest file, to the storage devices with the largest gap between their current allocated capacity and their bandwidth percentage. This ensures that high-performance storage devices with high read / write bandwidth percentages handle more file read tasks, while devices with low read / write bandwidth percentages handle relatively fewer files, thus achieving a match between storage device load and capacity.

[0036] This application uses the ratio of read and write bandwidth as the allocation basis to achieve the matching of storage device I / O capacity and load, so that the loading completion time of each storage device tends to be consistent, eliminating the bottleneck caused by a certain storage device, making full use of the actual performance of each storage device, ensuring the efficient use of high-bandwidth devices, thereby maximizing aggregate bandwidth in the concurrent loading stage and improving the loading efficiency of model weight files.

[0037] In one possible implementation of this application embodiment, when allocating multiple weight files to multiple storage devices according to the read / write bandwidth ratio and the file size of each weight file, the following methods can be used, but are not limited to: sorting the multiple weight files according to their respective file sizes to obtain sorted weight files; sequentially and cyclically allocating the sorted weight files to multiple storage devices so that the total weight file ratio of each storage device is the same as the read / write bandwidth ratio of each storage device; wherein, the total weight file ratio is the ratio of the total weight file allocated in the storage device to the total number of files, and the total number of files is the total number of multiple weight files of the model to be processed.

[0038] In the embodiments of this application, due to different model structures, the sizes of the weight files may vary. For example, weight files closer to the input or output layers may be larger, while weight files in intermediate layers may be relatively smaller. Therefore, these weight files need to be sorted, i.e., arranged in ascending or descending order according to file size, forming an ordered file list. The purpose of sorting is to ensure that, during the nighttime allocation process, large and small files are evenly distributed to various storage devices, avoiding the concentration of consecutive large files on a single storage device, which could lead to load imbalance.

[0039] Sequential cyclic allocation refers to allocating weighted files to storage devices one by one according to the sorted file order. After allocating to the last storage device, the process returns to the first storage device to continue allocating the next file, and so on, until all weighted files have been allocated. Simultaneously, it is necessary to ensure that the total number of weighted files stored on each storage device is equal to its read / write bandwidth ratio.

[0040] By sequentially distributing the sorted files in a cyclical manner, large and small files are alternately allocated to different devices, which makes the total number of files accumulated on each device approach the preset target ratio, that is, the same as the read and write bandwidth ratio.

[0041] This application sorts the weighted files and distributes them cyclically, so that the total weighted file volume carried by each storage device is basically consistent with its read / write bandwidth ratio. This achieves the matching of storage device I / O capacity and load, thereby improving the overall loading efficiency during the concurrent loading phase of weighted files.

[0042] In one possible implementation of this application embodiment, when generating storage path information corresponding to each of the multiple weight files, the following methods can be used, but are not limited to: recording the storage address corresponding to each of the multiple weight files, wherein the storage address is used to indicate the storage location of each of the multiple weight files in multiple storage devices; constructing a mapping relationship between the file identifiers corresponding to each of the multiple weight files and the storage address to obtain the storage path information.

[0043] In the embodiments of this application, since the weight files need to be distributed across multiple storage devices, the storage location of each weight file is no longer a fixed path, but is scattered across multiple storage devices on the server. Therefore, it is necessary to record the storage address of each weight file.

[0044] The storage address is a complete path description used to locate the specific storage location of the weight files. This storage address typically includes the mount point path of the storage device, followed by subdirectories and filenames, for example: / disk_b / weights / model-00001-of-00016.safetensors. Here, / disk_b represents a specific storage device, the subsequent path specifies the directory and filename on that storage device, and model-00001-of-00016.safetensors is the file identifier for the weight files.

[0045] A file identifier is a logical name or index used to uniquely identify each weight file, such as the filename of the weight file in the model to be processed (e.g., model-00001-of-00016.safetensors). A mapping relationship is a data structure that associates the file identifier of each weight file with its corresponding storage address. Mapping relationships can be implemented in various forms, such as key-value pairs, tables, or databases.

[0046] When loading model weight files, a file identifier is typically used to request the loading of a specific weight file. A mapping relationship allows for quick location and determination of the actual storage address corresponding to that file identifier, guiding the loading task to read the weight file from the correct storage device. The resulting storage path information is a dataset containing the mapping relationships between all weight file identifiers and their storage addresses. This storage path information is usually persistently stored on the server in the form of a manifest file (e.g., manifest.json).

[0047] This application records the storage address of each weighted file and constructs a mapping relationship between it and the file identifier. This enables the correct storage device and specific storage path to be quickly located based on the file identifier during the file loading stage, providing a scheduling basis for multi-threaded concurrent loading tasks and ensuring the correctness and reliability of concurrent file loading.

[0048] In one possible implementation of this application embodiment, when loading multiple weight files from multiple storage devices to the memory of the server in parallel, the following methods can be used, but are not limited to: reading storage path information, obtaining the mapping relationship between the file identifiers corresponding to each of the multiple weight files and the storage addresses corresponding to each of the multiple weight files based on the storage path information; creating a task thread pool, submitting the loading task to the task thread pool for parallel execution based on the mapping relationship, wherein each loading task thread in the task thread pool reads the weight file from its corresponding storage device through a preset file reading interface, and the preset file reading interface at least supports memory mapping functionality.

[0049] In the embodiments of this application, by reading the storage path information, the mapping relationship between the file identifier of each weight file and its storage address can be obtained. By obtaining this mapping relationship, the storage device where each weight file to be loaded is located and the specific path of the weight file on that storage device can be determined.

[0050] A task thread pool is a multi-threading management technique that pre-creates a certain number of worker threads to concurrently execute submitted loading tasks, thus avoiding the overhead of frequently creating and destroying threads. Based on the mapping relationship, each weight file read operation is encapsulated as an independent loading task, which includes the storage address of the weight file and other necessary read parameters. These loading tasks are then submitted to the task thread pool, which schedules idle threads to execute them in parallel.

[0051] Loading task threads refer to the worker threads in the thread pool that actually execute loading tasks. Each thread is responsible for reading one or more weighted files from a specified storage address. To ensure efficiency and compatibility, each loading task thread reads data through a preset file reading interface. The preset file reading interface is a custom-configured file reading interface, such as: the operating system's standard file I / O interface, a high-efficiency reading interface optimized for specific file formats (e.g., the memory-mapped `safetensors.torch.load_file` interface), or a wrapped reading function with concurrency control capabilities.

[0052] Through the preset file reading interface, the loading task threads can read weighted file data from their respective storage devices and load it into the server's memory.

[0053] This application achieves efficient parallel reading of weight files on multiple storage devices by creating a task thread pool and submitting loading tasks based on mapping relationships. The task thread pool not only avoids the resource overhead of frequently creating threads, but also provides concurrency control capabilities, which can dynamically adjust the degree of parallelism according to hardware resources and the number of tasks. Each loading task thread reads data from its corresponding storage device through a preset file reading interface, ensuring the stability and efficiency of the reading operation. It makes full use of the parallel processing capabilities of the processor and the aggregated I / O bandwidth of multiple storage devices, thereby shortening the loading time of model weight files and improving the startup efficiency of model services.

[0054] In one possible implementation of this application embodiment, when combining multiple weight files loaded into memory according to the original model structure order corresponding to the model to be processed, the following methods can be used, but are not limited to: transferring multiple weight files to a shared aggregation buffer, and combining multiple weight files according to the original model structure order corresponding to the model to be processed in the shared aggregation buffer to obtain the loaded weight files.

[0055] In the embodiments of this application, the shared aggregation buffer is a data storage area pre-allocated in memory that can be accessed simultaneously by multiple loading task threads. The buffer has the characteristics of sharing and aggregation. Sharing means that the buffer is open to all threads performing loading tasks, and each thread can write its loaded weight file into this buffer after completing the reading of its own weight file. Aggregation means that the buffer acts as a node for data aggregation, used to temporarily store all loaded weight files, providing complete data for subsequent combination operations.

[0056] The original model structure order corresponding to the model to be processed is usually determined by the model's configuration file and network definition, and is the basis for the model to be correctly recovered and run. In the shared aggregation buffer, based on the model's metadata information, the specific model location to which each weight file belongs is identified, and they are reassembled according to the original model structure order, ultimately forming a complete model state dictionary (state_dict) or an equivalent data structure. The loaded weight file obtained after sequential combination can be directly accepted by the model loader to initialize model parameters, thereby completing the entire model loading process.

[0057] This application uses a shared aggregation buffer as the central node for data aggregation, providing a data writing target for multiple concurrently loaded threads. This avoids the chaos and combination difficulties caused by data dispersion. The weight files are combined in the shared aggregation buffer according to the original model structure order, ensuring that the dispersed and concurrently loaded weight files can be accurately restored to the complete parameter set required by the model, thus guaranteeing the correctness of model loading.

[0058] In one possible implementation of this application embodiment, when reading storage path information, it can be implemented in, but is not limited to, the following ways: by converting the storage path information into a separator string containing the storage addresses corresponding to each of the multiple weight files for reading; or by converting the storage path information into a virtual single model path for reading, wherein the virtual single model path maintains symbolic links pointing to weight files in multiple storage devices.

[0059] In the embodiments of this application, the reading of storage path information can be achieved through at least two optional interface modes.

[0060] Specifically, the first method involves reading the data using a delimiter string. In this method, the storage path information is a set of metadata recording the storage locations of all weight files, containing a mapping between the file identifier of each weight file and its storage address. For ease of use, this mapping is converted into a delimiter string. The delimiter string is a single string containing multiple path information entries, with each storage address separated by a specific delimiter (e.g., comma, semicolon, or vertical bar). During model loading, the upper-level framework typically accepts a model path parameter. By directly passing this delimiter string as the model path parameter, the upper-level framework can obtain the actual storage addresses of all weight files at once. The actual storage address is an absolute path directly pointing to the actual stored weights. A collection of multiple paths can form a complete weight file. However, implementing this first method requires appropriate modifications to the framework's path parsing code.

[0061] The second approach involves reading through a virtual single model path. In this method, a virtual single model path is created in the server's file system. This virtual single model path appears as a regular directory, and all weight files are stored under this single model path for the upper-level framework. However, this single model path does not actually store the weight files; instead, it maintains symbolic links pointing to the actual weight files on various storage devices. These symbolic links contain path references to other files or directories. When accessing a weight file through a symbolic link, it automatically redirects to the target file it points to. During model loading, the upper-level framework only needs to specify this virtual single model path. When attempting to read a weight file under this model path, it will automatically redirect to the file actually stored on multiple storage devices via the symbolic link. The single model path maintains a direct correspondence with the upper-level framework, requiring no modification to the path resolution code and adapting to the current upper-level framework. However, the actual weight files within the single model path are stored on different hard drives via symbolic links, requiring the corresponding hard drives to be mounted when creating the container.

[0062] This application uses a delimiter string to merge multiple storage addresses into a single parameter, achieving minimal modification to existing frameworks and making it suitable for scenarios that allow custom parameter parsing. By using a virtual single model path and symbolic links, it provides a transparent view to upper-layer frameworks, achieving zero-modification integration with existing frameworks. The framework does not need to be aware of the actual distribution of underlying files to achieve accelerated multi-disk concurrent loading. This effectively solves the interface adaptation problem between concurrent loading and existing frameworks, improving the practicality and versatility of the solution.

[0063] In one possible implementation of this application embodiment, the following method may also be used, but is not limited to: monitoring the weight loading information corresponding to each of the plurality of storage devices; verifying whether there is an anomaly in the weight file loaded from the plurality of storage devices according to the verification value in the weight loading information; in response to the anomaly in the target weight file loaded from the target storage device, loading the target weight file from other storage devices according to the priority order of the read / write bandwidth ratio from high to low, until the target weight file without anomalies is loaded, wherein the target storage device is any one of the plurality of storage devices.

[0064] In the embodiments of this application, to improve reliability and fault tolerance, a complete copy of the model weight file can be maintained on each storage device. That is, the complete content of all weight files is independently stored on each hard disk, forming a highly reliable redundant backup. The weight loading information refers to various status data and metadata generated during the weight file reading process, including but not limited to the actual read path of each weight file, the number of bytes read, the loading time, and the checksum used for integrity verification.

[0065] Furthermore, the checksum refers to a fixed-length hash value calculated from the content of the weight file using a specific algorithm (e.g., the SHA256 checksum algorithm). During the verification process, the content of the weight file loaded from the storage device is read, automatically segmented according to preset rules, and its checksum is calculated or checked segment by segment. Anomalies refer to various problems that may occur during the storage or retrieval of the weight file, including but not limited to checksum mismatch caused by corrupted file data (i.e., the calculated checksum is inconsistent with the pre-recorded standard checksum), input / output errors occurring during file retrieval, missing parts of the file content, or format errors. When the checksum comparison fails or other retrieval anomalies are detected, the weight file is determined to be abnormal and cannot be directly used for model loading.

[0066] The target storage device refers to the storage device currently attempting to read a specific weighted file during concurrent loading; it can be any one of multiple storage devices. The target weighted file refers to a specific weighted file that is detected as having an anomaly during loading on the target storage device. When an anomaly occurs, automatic failover and recovery are initiated. Specifically, the read / write bandwidth ratios of all other storage devices (i.e., all storage devices other than the target storage device) are obtained. Following the order of read / write bandwidth ratio from highest to lowest, i.e., starting with the storage device with the highest performance, the same target weighted file is attempted to be loaded from other storage devices sequentially.

[0067] Because each storage device maintains a complete redundant backup of the weight file, valid copies of the target weight file exist on other devices. In each attempt, the newly loaded weight file undergoes the same checksum verification. If the verification passes and there are no other anomalies, the weight file is considered error-free and used for subsequent model combination loading. If anomalies still exist, the process continues, switching to the next highest-performing storage device according to priority, and so on, until an error-free target weight file is successfully loaded.

[0068] In one possible example, during the weight file loading process, it is necessary to configure highly reliable redundant backups as needed (keeping a complete copy of the weight file on each storage device) and dynamically monitor the loading status of the weight files on each storage device. By automatically pruning multiple small segments of verification values, if the verification values ​​are found to be mismatched or the weight file loading is abnormal, the system will automatically switch to other storage devices to read the weight file, starting from the storage device with the highest performance (i.e., the storage device with the highest read / write bandwidth ratio) to ensure that the model loads normally.

[0069] In one possible implementation of this application embodiment, after combining multiple loaded weight files according to the original model structure order corresponding to the model to be processed to obtain a loaded weight file, the following methods can also be used, but are not limited to: caching the metadata or data pointer of the loaded weight file based on the least recently used strategy; and restoring the loaded weight file based on the cached metadata or data pointer in response to a subsequent hot-start model loading request.

[0070] In the embodiments of this application, after the initial cold start loading is completed, the loaded weight file is fully present in the server memory and can be further transferred to the GPU memory for model calculation. At this time, the relevant information of the loaded weight file is cached. The caching operation is managed based on the Least Recently Used (LRU) strategy. By adopting the LRU strategy, the cache always retains the hot data that is most likely to be used again, thereby improving the cache hit rate.

[0071] The cached content includes metadata or data pointers to the loaded weight file. Metadata refers to information describing the attributes of the weight file but not containing the complete weight data itself, such as the directory structure, file list, and storage address mapping of the weight file. Data pointers are references or handles pointing to the actual location of the weight file in memory, such as memory addresses, file descriptors, or object references.

[0072] A warm-start model loading request refers to a loading instruction issued when restarting the same model service or resuming the same model training after an initial cold start. In a warm-start scenario, the server may have just completed a service stop or restart, but some data may still be retained. When a warm-start request is received, the cache is first checked for previously stored metadata or data pointers. If valid metadata and / or data pointers exist, this information can be used to directly restore the weight file from memory without initiating a complete loading process again.

[0073] The cache space refers to the memory capacity used for cached data. The first preset space threshold is a pre-set value, for example, when the available cache space reaches a certain percentage of the total cache capacity (e.g., 80%), a cache cleanup operation is triggered. The second preset space threshold is a target value higher than the first preset space threshold, representing the safe level to which the cache space needs to be restored, for example, when the available cache space is greater than 60% of the total capacity.

[0074] When cache space is tight, deletion is performed in descending order of priority based on the read / write bandwidth ratio of storage devices. That is, metadata or data pointers of weighted files loaded from the highest-performance storage device (highest read / write bandwidth ratio) are deleted first, and cached content loaded from the lowest-performance storage device (lowest read / write bandwidth ratio) is deleted last.

[0075] In subsequent warm restarts or reload scenarios, if the cache has been partially cleared, the deleted weight files need to be reloaded. In this case, since high-performance storage devices have faster read speeds, reloading the data is less costly; therefore, the cache of high-performance storage devices is cleared first. Conversely, low-performance storage devices have slower read speeds, and reloading them is more costly; therefore, cached data on low-performance storage devices should be retained as much as possible.

[0076] The change in metadata or data pointers refers to modifications to the model weight file itself, such as model version updates, weight file overwriting or replacement, or changes in file checksums. Since the metadata or data pointers stored in the cache are generated based on older versions of the weight file, changes to the weight file will result in inconsistencies between the cached content and the actual data, rendering it unusable for warm-start recovery and potentially leading to model loading errors. Therefore, upon detecting any change, a cold-start model loading command must be executed. This involves performing a complete cold-start loading process, including hardware detection, intelligent weight distribution, and concurrent loading across multiple hard drives, to ensure that the latest weight file is loaded.

[0077] In one possible example, when the weight file itself is modified, a cold start is required to ensure cache consistency; when the cache is tight, weights loaded from the highest-performance storage device are deleted first, followed by weights from the lowest-performance storage device, thus ensuring the efficiency of reloading.

[0078] It should be noted that the weight loading method for each node is the same when the weights of multiple nodes are divided.

[0079] This application caches the metadata or data pointers of loaded weight files, enabling subsequent hot-start processes to avoid repeatedly executing file loading operations, thus shortening the repeated startup time of the model service and improving its response speed and availability. An LRU (Least Recently Used) strategy is employed for cache management, ensuring that the cache always retains the most likely frequently used data, improving the utilization efficiency of cache resources.

[0080] In one possible implementation of this application embodiment, to facilitate understanding of the implementation process of this application embodiment, this application embodiment also provides a model file loading process topology diagram, such as... Figure 2 As shown, it includes the following: Weighted file intelligent distribution stage (preprocessing): Hardware detection: Run tools to detect the continuous read and write bandwidth of each hard drive on the server.

[0081] Weight allocation: Based on the bandwidth ratio of each hard drive, the total weight file set of the model to be loaded is split and allocated. For example, if the bandwidth ratio of the three hard drives is 4:3:2, the weight files are sorted by size and then distributed to each hard drive in a cyclical manner, so that the amount of data carried by each hard drive is roughly proportional to its I / O capacity, avoiding the "weakest link" effect.

[0082] Metadata Recording: Generate a global manifest file that records the mapping relationship between each weight file and its actual storage path.

[0083] Multi-disk concurrent loading phase (runtime): Loader initialization: Modifies or encapsulates the framework's weighted loading modules. The loader reads the manifest file and constructs a task mapping table of file paths and storage disks.

[0084] Task scheduling: The loader creates a thread pool. Based on the task mapping table, it submits tasks for reading weighted files located on different physical hard drives to the thread pool for concurrent execution. The key point is to ensure that the number of tasks sent to the same hard drive simultaneously does not exceed its optimal concurrent queue depth, while tasks sent to different hard drives are executed in full parallelism.

[0085] Concurrent reads and memory mapping: Each worker thread uses an efficient file read interface to read the assigned weighted file. Memory mapping technology allows file data to be loaded on demand by page cache, reducing the instantaneous pressure on physical memory.

[0086] Data aggregation: The weight tensor data read by each thread is sent to a shared aggregation buffer. The loader is responsible for reassembling the tensors from different files into a complete model state dictionary according to the original order of the model structure.

[0087] Caching and warm-start optimization: The loader integrates a smart caching layer (e.g., based on an LRU strategy). After the initial cold start, pointers or metadata of weighted data can be cached. When a service is warm-started, the loader first checks the cache validity and prioritizes restoring data from the operating system's page cache or its own cache to avoid duplicate disk I / O.

[0088] Interface adaptation: To ensure compatibility with existing frameworks, two interface modes are provided: Multi-path mode: Pass multiple paths separated by delimiters to the model parameters of the framework, and the loader internally parses and performs concurrent loading.

[0089] Single-path virtualization mode: Maintains a single virtual model path, under which all major files are symbolic links pointing to the actual files on each hard drive. It is transparent to the framework, and concurrent access is implemented by the underlying loader.

[0090] Once all weight files have finished loading, the model service should start normally.

[0091] In summary, the embodiments of this application can achieve at least the following technical effects: 1. By aggregating the I / O bandwidth of multiple hard drives, the loading process is changed from serial to parallel. The distributed reading of multiple hard drives has the advantages of bandwidth aggregation, high concurrency, and high scalability, which can perform high-performance loading of large model weights.

[0092] 2. No need to invest in expensive high-end storage equipment; existing server idle hard drive resources can be fully utilized through software solutions, achieving cost reduction and efficiency improvement.

[0093] 3. It fully leverages the thread scheduling capabilities of multi-core processors and the parallel I / O capabilities of multiple hard drives, thereby improving the overall utilization of hardware resources.

[0094] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0095] Embodiments of this application also provide a model file processing apparatus. Figure 3 A schematic diagram of the structure of a model file processing device provided in this application is shown below. Figure 3 As shown, it includes: The allocation unit 31 is used to allocate multiple weight files of the model to be processed to multiple storage devices according to the read and write bandwidth of each of the multiple storage devices in the server. The generation unit 32 is used to generate storage path information corresponding to each of the multiple weight files, wherein the total number of weight files allocated to the multiple storage devices is positively correlated with the read and write bandwidth of each of the multiple storage devices. Loading unit 33 is used to respond to model loading instructions and, based on storage path information, execute loading tasks in parallel from multiple storage devices to the memory of the server. The number of loading tasks executed in parallel on the same storage device is less than the preset concurrent queue depth, and loading tasks are executed in parallel on different storage devices. Combination unit 34 is used to combine multiple weight files loaded into memory according to the original model structure order corresponding to the model to be processed, so as to obtain the loaded weight file.

[0096] In one embodiment of this application, the allocation unit 31 is specifically used for: The ratio of the read / write bandwidth of each of the multiple storage devices to the total bandwidth is calculated to obtain the read / write bandwidth percentage of each of the multiple storage devices. The total bandwidth is the sum of the read / write bandwidth of each of the multiple storage devices. Based on the read / write bandwidth ratio and the file size of each weighted file, multiple weighted files are allocated to multiple storage devices. The higher the read / write bandwidth ratio of a storage device, the more weighted files are allocated.

[0097] In one embodiment of this application, the allocation unit 31 is specifically used for: The multiple weight files are sorted according to their respective file sizes to obtain the sorted weight files; The sorted weighted files are sequentially and cyclically distributed to multiple storage devices, so that the total weighted file proportion of each storage device is the same as the read / write bandwidth proportion of each storage device. The weight file total percentage is the ratio of the total number of weight files allocated in the storage device to the total number of files, and the total number of files is the total number of weight files of the model to be processed.

[0098] In one embodiment of this application, the generation unit 32 is specifically used for: Record the storage address corresponding to each of the multiple weight files, where the storage address is used to indicate the storage location of each weight file in multiple storage devices; Construct a mapping relationship between the file identifiers of multiple weighted files and their respective storage addresses to obtain storage path information.

[0099] In one embodiment of this application, the loading unit 33 is specifically used for: Read the storage path information and obtain the mapping relationship between the file identifiers of multiple weight files and their respective storage addresses based on the storage path information; A task thread pool is created, and loading tasks are submitted to the task thread pool for parallel execution based on the mapping relationship. Each loading task thread in the task thread pool reads a weighted file from its corresponding storage device through a preset file reading interface, which at least supports memory mapping functionality.

[0100] In one embodiment of this application, the combining unit 34 is specifically used for: Multiple weight files are transferred to a shared aggregation buffer. In the shared aggregation buffer, the multiple weight files are combined in the order of the original model structure corresponding to the model to be processed to obtain the loaded weight file.

[0101] In one embodiment of this application, the loading unit 33 is specifically used for: The storage path information is read by converting it into a separator string containing the storage addresses of each of the multiple weight files; Alternatively, storage path information can be read by converting it into a virtual single model path, where the virtual single model path maintains symbolic links pointing to weight files in multiple storage devices.

[0102] In one embodiment of this application, the loading unit 33 is further configured to: Monitor the weight loading information corresponding to each of the multiple storage devices; The weight files loaded from the multiple storage devices are checked for anomalies based on the verification value in the weight loading information. In response to an anomaly in the target weight file loaded from the target storage device, the target weight file is loaded from other storage devices according to the priority order of read / write bandwidth ratio from high to low, until the target weight file without anomalies is loaded, wherein the target storage device is any one of the plurality of storage devices.

[0103] In one embodiment of this application, the loading unit 33 is specifically used for: Based on the least recently used strategy, cache the metadata or data pointers of the weight files that have been loaded. In response to subsequent hot-start model loading requests, the loaded weight file is restored based on the cached metadata or data pointers; In response to the cache space being less than a first preset space threshold, the metadata or data pointers of the weighted files loaded from the multiple storage devices are deleted in order of priority from high to low according to the read / write bandwidth ratio, until the cache space is greater than a second preset space threshold. In response to the detection of any change in metadata or data pointer, execute the cold start model loading instruction.

[0104] For a description of the features in the embodiment corresponding to the model file processing device, please refer to the relevant description in the embodiment corresponding to the model file processing method, which will not be repeated here.

[0105] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the model file processing method.

[0106] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the model file processing method.

[0107] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0108] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model file processing method embodiments.

[0109] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model file processing method embodiments.

[0110] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be executed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] The foregoing has provided a detailed description of a model file processing method and electronic device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for processing model files, characterized in that, include: Based on the read / write bandwidth of each of the multiple storage devices in the server, multiple weight files of the model to be processed are allocated to the multiple storage devices, and storage path information corresponding to each of the multiple weight files is generated. The total number of weight files allocated to the multiple storage devices is positively correlated with the read / write bandwidth of each of the multiple storage devices. The step of allocating multiple weight files of the model to be processed to the multiple storage devices includes: allocating the multiple weight files to the multiple storage devices according to the read / write bandwidth ratio of each storage device and the file size of each weight file, wherein the higher the read / write bandwidth ratio of the storage device, the more weight files are allocated. In response to the model loading instruction, according to the storage path information, loading tasks are executed in parallel from the multiple storage devices to the memory of the server. The number of loading tasks executed in parallel on the same storage device is less than the preset concurrency queue depth, and loading tasks are executed in parallel on different storage devices. The multiple weight files loaded into the memory are combined in the order of the original model structure corresponding to the model to be processed to obtain the loaded weight file; Cache the metadata or data pointers of the loaded weight file; In response to a subsequent hot-start model loading request, the loaded weight file is restored based on the metadata or data pointers in the cache.

2. The method for processing model files according to claim 1, characterized in that, The method further includes: The ratio of the read / write bandwidth of each of the plurality of storage devices to the total bandwidth is calculated to obtain the read / write bandwidth percentage of each of the plurality of storage devices, wherein the total bandwidth is the sum of the read / write bandwidth of each of the plurality of storage devices.

3. The method for processing model files according to claim 2, characterized in that, The step of allocating the multiple weighted files to the multiple storage devices according to the read / write bandwidth ratio and the file size corresponding to each of the multiple weighted files includes: The multiple weight files are sorted according to their respective file sizes to obtain the sorted weight files; The sorted weighted files are sequentially and cyclically distributed to the multiple storage devices, such that the total weighted file ratio of each storage device is the same as the read / write bandwidth ratio of each storage device. The percentage of the total weight files is the ratio of the total number of weight files allocated in the storage device to the total number of files, and the total number of files is the total number of multiple weight files of the model to be processed.

4. The method for processing model files according to claim 1, characterized in that, The storage path information corresponding to each of the multiple weight files includes: Record the storage address corresponding to each of the plurality of weight files, wherein the storage address is used to indicate the storage location of each of the plurality of weight files in the plurality of storage devices; The storage path information is obtained by constructing a mapping relationship between the file identifiers corresponding to each of the multiple weighted files and the storage addresses.

5. The method for processing model files according to claim 1, characterized in that, The step of executing the loading task in parallel from the multiple storage devices to the server memory based on the storage path information includes: Read the storage path information, and obtain the mapping relationship between the file identifiers corresponding to the multiple weight files and the storage addresses corresponding to the multiple weight files based on the storage path information; A task thread pool is created, and the loading task is submitted to the task thread pool for parallel execution based on the mapping relationship. Each loading task thread in the task thread pool reads a weighted file from its corresponding storage device through a preset file reading interface, and the preset file reading interface at least supports memory mapping functionality.

6. The method for processing model files according to claim 1, characterized in that, The process of combining the multiple weight files loaded into the memory according to the original model structure order corresponding to the model to be processed to obtain the loaded weight files includes: The multiple weight files are transferred to a shared aggregation buffer. In the shared aggregation buffer, the multiple weight files are combined in the order of the original model structure corresponding to the model to be processed to obtain the loaded weight file.

7. The method for processing model files according to claim 5, characterized in that, Reading the storage path information includes: The storage path information is read by converting it into a separator string containing the storage addresses corresponding to the multiple weighted files. Alternatively, the storage path information can be converted into a virtual single model path for reading, wherein the virtual single model path maintains symbolic links pointing to weight files in the multiple storage devices.

8. The method for processing model files according to claim 2, characterized in that, The method further includes: Monitor the weight loading information corresponding to each of the multiple storage devices; The weight files loaded from the multiple storage devices are checked for anomalies based on the verification value in the weight loading information. In response to an anomaly in the target weight file loaded from the target storage device, the target weight file is loaded from other storage devices according to the priority order of read / write bandwidth ratio from high to low, until the target weight file without anomalies is loaded, wherein the target storage device is any one of the plurality of storage devices.

9. The method for processing model files according to claim 2, characterized in that, After combining the loaded weight files according to the original model structure order corresponding to the model to be processed to obtain the loaded weight files, the method further includes: In response to the cache space being less than a first preset space threshold, the metadata or data pointers of the weighted files loaded from the multiple storage devices are deleted in order of priority from high to low according to the read / write bandwidth ratio, until the cache space is greater than a second preset space threshold. In response to the detection of any change in metadata or data pointer, execute the cold start model loading instruction.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the model file processing method as described in any one of claims 1 to 9.

Images