Large model file rapid loading method and system under Linux system
By reserving a contiguous physical memory area during Linux system startup and prioritizing the loading of large model files, the problem of slow loading speed of large model files is solved, achieving more efficient memory utilization and improved user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-03-27
AI Technical Summary
On Linux systems, large model files load slowly, resulting in a poor user experience. This is mainly due to performance bottlenecks caused by discontinuous physical memory and increased disk request queue waiting time.
Reserve a contiguous physical memory area when the system starts up, and map it to the application's virtual memory space when loading large model files. Prioritize loading large model files and dynamically release memory for other modules when memory is insufficient. Add a priority scheduling class for large model files to the IO scheduler to improve read and write priority.
It significantly reduces the latency of loading large model files, improves loading efficiency and system resource utilization, and enhances user experience and system stability.
Smart Images
Figure CN121742933A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of large model file, and particularly relates to a large model file fast loading method and system under a Linux system. BACKGROUND
[0002] When using AI applications on an operating system, a large model file usually needs to be loaded from a file system to a memory first, and then it is decided whether to copy to the exclusive memory of a GPU / NPU according to the hardware environment of the system, as shown in the process of Figure 1 When a user starts to use a large model, an AI application (taking the ollama inference framework as an example) first allocates memory from a memory management module of a Linux kernel, usually a dma_buf, then loads a large model file to a page cache of a file system through a file system, and finally copies the content of the page cache to the dma_buf. If a GPU / NPU integrated in a CPU is used for inference of the AI application, the integrated GPU / NPU directly uses the large model file in the dma_buf to start the inference process, and if an independent GPU / NPU is used, the dma engine is used to transmit the large model file in the dma_buf to the memory of the independent GPU / NPU.
[0003] With the increasing size of the parameters of a large model, the large model file is also getting larger, Figure 2 is the amount of memory required by some large model files. As can be seen from Figure 2 , it takes a certain amount of time to load such a large file into the memory, for example, a desktop computer with an intel i5 processor, 16G ddr, and a sata interface SSD hard disk. Using ollama to load a deepseek-r1:7b, a large model file of 4.7G in size, takes about 10s, which affects the user experience. As shown in Figure 3 , the traditional large model file storage and reading process can be seen to have the following performance bottlenecks: 1. Since the system usually does not have a large block of physical memory to allocate, the storage of the large model file is usually scattered in several non-continuous sectors of the file system. In the process of reading the file, finding the information of these sectors will increase a lot of performance overhead, and the larger the file, the more overhead required; 2. Each non-continuous sector will form a request, and the more dispersed the file storage is, the more requests there will be, and the more the performance will decrease; 3. Reading each continuous sector can form a disk request, and these requests will be queued in the disk request queue together with the read-write disk requests of other applications, and the waiting time is an additional overhead. SUMMARY
[0004] The technical problem solved by the present application: In view of the above problems of the prior art, a large model file fast loading method and system under a Linux system are provided to improve the loading speed of large model files and improve user experience.
[0005] To solve the above technical problems, the technical scheme adopted by the present application is: A large model file fast loading method under a Linux system, comprising the steps of: When the Linux system starts, a continuous physical memory area of a preset size for loading a large model file is reserved according to the size of the large model file to be read; When an application program in the Linux system needs to load a large model file, the reserved continuous physical memory area is mapped to the virtual memory address space of the application program, and the large model file is preferentially loaded into the continuous physical memory area, and the memory area occupied by the large model file is released after the large model file is used up; When other modules in the Linux system request memory allocation, if it is detected that the continuous physical memory area is currently not loaded with a large model file and the sum of the requested memory size and the size of the memory already allocated in the continuous physical memory area is less than a preset threshold, then allocate the requested memory size to the corresponding module in the continuous physical memory area, otherwise do not allocate.
[0006] Further, the method further comprises: If it is detected that a large model file needs to be loaded and the continuous physical memory is insufficient for allocation, then release all the memory allocated to other modules of the Linux system in the continuous physical memory area for use by the large model file.
[0007] Further, the continuous physical memory area of a preset size for loading a large model file is reserved according to the size of the large model file to be read, specifically comprising: Configure the reserved memory area size and memory start address in the kernel startup parameter, or configure the reserved memory area size and memory start address and end address in the device tree to reserve a continuous physical memory area of a preset size.
[0008] Further, the large model file is preferentially loaded into the continuous physical memory area, specifically comprising: A large model file scheduling class with a priority between the real-time task scheduling class and the normal application scheduling class is added in the IO scheduler of the Linux system; When an application needs to load a large model file, multiple read-write requests are initiated according to a large model partition where the large model file is located, with a preset length, and the scheduling classes corresponding to the multiple read-write requests are set as the large model file scheduling class, so that the read-write priority of the large model file during loading is higher than the read-write priority of ordinary applications. The IO scheduler sends all read-write requests to the disk driver in order of priority, so that the disk driver writes the large model file into the continuous physical memory area according to the multiple read-write requests.
[0009] Further, the preset length is 128M, corresponding to 250000 sectors.
[0010] Further, the preset threshold satisfies the following conditions: When the large model file starts to be loaded, the total time consumed by the AMA memory management module for releasing the allocated memory in the continuous physical memory area is less than the total time consumed by the Linux memory manager dma buf for releasing memory, wherein the AMA memory management module is used to manage the continuous physical memory area.
[0011] Further, the preset threshold satisfies the following formula: T reclaim (AMA size · ratio)<T allocate dma_buf (AMA size) In the above formula, AMA size is the total memory size of the continuous physical memory area, ratio is the preset threshold, T allocate dma_buf () is the total time consumed by dma buf for releasing memory, and T reclaim () is the total time consumed by the AMA memory management module for releasing memory.
[0012] Further, the large model file is pre-stored in a large model partition of a disk, and is stored in a continuous sector order in the large model partition.
[0013] Further, when mapping the reserved continuous physical memory area to the virtual memory address space of the application, mmap is specifically used for mapping, and the steps are as follows: Traverse each bit in the bitmap array corresponding to the continuous physical memory area and determine whether it is 0, if not 0, further determine the memory page type, if it is an anonymous page, take the swap partition mode, if it is a dirty page, write back to the file system or discard processing, to realize the recycling or migration of memory; After recycling or migration is completed, the bitmap array is emptied, the base_addr is mapped to the virtual memory address space of the application, and the virtual address is returned.
[0014] A large model file fast loading system under a Linux system, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to perform a large model file fast loading method under a Linux system.
[0015] Compared with the prior art, the present application has the advantages that: The present application reserves a continuous physical memory area during the system startup phase, ensuring that the large model file does not need to go through complex memory fragmentation during loading, significantly reducing the loading delay caused by non-continuous physical memory, and improving the startup speed of the large model. By mapping the reserved physical memory area to the application virtual address space and prioritizing it for loading the large model, the overall efficiency and reliability of the loading can be improved. During the loading of the large model, the reserved area is dynamically allocated to other system modules, which can improve the utilization of system memory resources and avoid long-term vacancy. The present application can achieve a balance between large model loading speed and system resource utilization, improving the running performance and stability of the Linux system in the large model application scenario. BRIEF DESCRIPTION OF DRAWINGS
[0016] Figure 1 A flowchart for loading a large model file from a file system to a memory in an existing AI application.
[0017] Figure 2 A schematic diagram of the amount of memory required for some large model files.
[0018] Figure 3 A flowchart of the entire process of reading a large model file by an AI application.
[0019] Figure 4 A flowchart of the large model file fast loading method under the Linux system of the embodiment of the present application.
[0020] Figure 5 A schematic diagram of the IO scheduler queue in the specific application embodiment.
[0021] Figure 6 A flowchart of the entire process of reading a large model file by an AI application in the specific application embodiment.
[0022] Figure 7 A flowchart of the allocation of continuous memory to other system modules when no large model file is loaded in the specific application embodiment.
[0023] Figure 8 A flowchart of the memory area mapping and large model file loading by calling mmap in the specific application embodiment.
[0024] Figure 9 A schematic diagram of the AI application virtual memory layout after mmap mapping in the specific application embodiment.
[0025] Figure 10 Flow chart for releasing memory in specific application examples. DETAILED DESCRIPTION
[0026] In order to better understand the above technical solutions, the above technical solutions will be described in detail below in combination with the drawings in the specification and specific embodiments.
[0027] The abbreviations and key terms of the present application are defined as follows: LLM: large language model, large language model.
[0028] CMA: contiguous memory allocator, one of the linux memory managers, manages large blocks of contiguous physical memory.
[0029] DMA (Direct Memory Access) is a data transfer technology in computer systems that allows peripherals (such as hard drives, network cards, graphics cards, etc.) to exchange data directly with memory without the involvement of the CPU, thereby improving system performance and reducing CPU burden.
[0030] HugeTLB Pages: Static Large Pages, one of the linux virtual memory management methods.
[0031] THP: Transparent Huge Pages, one of the linux virtual memory management methods.
[0032] dma buf: one of the linux memory managers, used to manage cross-subsystem and cross-driver shared DMA buffers.
[0033] dma engine: a hardware device used to copy data between memory and device registers.
[0034] pagecache: memory used by the linux kernel file system, the kernel first copies the file system content to the pagecache, and then copies it to the application.
[0035] AMA: AI memory allocator, the linux kernel provided by the present application is customized for AI memory manager.
[0036] GPU: Graphics processing unit, graphics card, but also for AI-related calculations.
[0037] NPU: Neural network processing unit, used for AI-related calculations.
[0038] GGUF: General Generative Unified Format, a storage format for large language model files.
[0039] like Figure 4 As shown in this embodiment, the method for fast loading of large model files under the Linux system includes the following steps: When the Linux system starts, it reserves a contiguous physical memory area of a preset size (AMA size) for loading large model files as needed. When an application in a Linux system needs to load a large model file, the reserved contiguous physical memory region is mapped to the application's virtual memory address space, and the large model file is loaded into the contiguous physical memory region first. After the large model file is finished being used, the memory region occupied by the large model file is released. When other modules in the Linux system request memory allocation, if it is detected that the contiguous physical memory region is not currently loaded with a large model file and the sum of the requested memory size and the allocated memory size in the contiguous physical memory region is less than a preset threshold (ratio), then the requested memory size is allocated to the corresponding module in the contiguous physical memory region; otherwise, no allocation is made.
[0040] Specifically, this embodiment provides a Linux kernel memory management module AMA (AI memory allocator). This module requests and reserves a contiguous segment of physical memory when the system starts up to load large model files. By reserving contiguous physical memory, the number of times AI applications request memory can be reduced, thereby improving efficiency.
[0041] The AMA memory management module uses the following information to manage allocated physical memory: base_addr: A 64-bit unsigned integer, representing the physical address of the memory region managed by the AMA memory management module.
[0042] count: A 64-bit unsigned integer representing the number of memory region pages managed by the AMA memory management module.
[0043] bitmap: A 64-bit unsigned integer array that shows the usage of pages in the memory region managed by the AMA memory management module. Each bit indicates whether a page has been allocated or used.
[0044] ratio: 64-bit unsigned integer data, a limit ratio used by the memory region managed by the AMA memory management module.
[0045] llm_loaded: Boolean data, indicating whether the memory region managed by the AMA memory management module has loaded a large model file.
[0046] In addition, the AMA memory management module also provides several functions: mmap: used to map the memory region managed by the AMA memory management module to the virtual memory space of the AI application, The AI application can copy the large model file from the file system to the address.
[0047] ama_alloc: when the memory region managed by the AMA memory management module does not load a large model file, other kernel modules can call this function to allocate contiguous physical memory.
[0048] ama_free: after the kernel module uses the memory allocated by ama_alloc, call this function to return the memory to the AMA memory management module.
[0049] In this embodiment, the method further comprises: If it is detected that the large model file needs to be loaded and the contiguous physical memory is insufficient for allocation, all the memory allocated to other modules of the Linux system in the contiguous physical memory region is released for use by the large model file. It can be understood that by automatically recycling the memory occupied by other modules in this region before loading the large model, a priority guarantee mechanism for large model loading requirements can be achieved, avoiding resource conflicts.
[0050] In this embodiment, a contiguous physical memory region of a preset size for loading a large model file is reserved according to the size of the large model file that needs to be read, specifically comprising: Reserve a contiguous physical memory region of a preset size by configuring the reserved memory region size and memory start address in the kernel startup parameter, or configuring the reserved memory region size and memory start address and end address in the device tree.
[0051] In a specific application embodiment, the AMA memory management module can reserve contiguous physical memory through kernel startup parameters or device trees. The format of the startup parameter is: ama=size@base_addr The device tree format is: reserved-memory { #address-cells =<1>; #size-cells =<1>; ranges; / * Address must be kept in the lower 256 MiBs of DRAM for VE. * / default-pool { compatible = "shared-dma-pool"; size =<0x<6000000>; alloc-ranges =<0x40000000 0x10000000>; reusable; linux,cma-default; }; }; For example, load deepseek-r1:7b, AMA needs the system to reserve 5G continuous memory for it, which can be set in the kernel startup parameter as ama=5G@0X80000000.
[0052] The AMA memory management module reserves the area for use according to the setting of the kernel startup parameter, and calls the memblock_reserve function provided by the Linux kernel.
[0053] In this embodiment, the large model file is preferentially loaded into the continuous physical memory area, specifically including: A large model file scheduling class with a priority between the real-time task scheduling class and the ordinary application scheduling class is added in the IO scheduler of the Linux system; When the application needs to load the large model file, multiple read-write requests are initiated according to the large model partition where the large model file is located with a preset length, and the scheduling class corresponding to the multiple read-write requests is set as the large model file scheduling class, so that the read-write priority of the large model file during loading is higher than the ordinary application read-write priority; The IO scheduler sends all read-write requests to the disk driver in order of priority, so that the disk driver writes the large model file into the continuous physical memory area according to the multiple read-write requests.
[0054] In a specific application embodiment, the IO scheduler queue is as shown in Figure 5 Each IO scheduling class has a read file queue, and a new scheduling class IOPRIO_CLASS_AI is added, whose priority is between IOPRIO_CLASS_RT (real-time task) and IOPRIO_CLASS_BE (ordinary application), so that the AI read file demand is prior to the ordinary application.
[0055] The specific process for reading large model files is as follows: Figure 6 As shown, a separate partition (AIpartition) is created on the disk, and the large model file is stored in this partition in consecutive sectors; When reading the file, the large model file is constructed into a bio in units of 128M (that is, 250,000 sectors); A new scheduling class IOPRIO_CLASS_AI has been added to the IO scheduler to increase the priority of AI application requests, giving them priority over ordinary applications in disk read and write processing. The rest of the process is the same as reading and writing ordinary files.
[0056] In a specific application embodiment, the process of other modules in the Linux system requesting memory allocation is as follows: Figure 7 As shown, using ama_alloc, when the AMA memory does not load a large model file, other modules of the system can call this function to allocate contiguous memory. If the AMA memory region has such a large contiguous memory and the total amount of memory already allocated in the region plus the newly requested memory does not exceed the ratio, allocation can be made and a pointer to a physical page is returned. If the above conditions are not met, allocation cannot be made and a null pointer NULL is returned.
[0057] The ratio setting is an empirical value. When AI starts requesting and using AMA region memory, the AMA memory module needs to migrate and reclaim memory from other modules. The time consumed should not exceed that of the traditional dma_buf allocation method. In other words, the preset threshold ratio satisfies the following conditions: When a large model file begins loading, the total time taken to release allocated memory in the contiguous physical memory region via the AMA memory management module is less than the total time taken to release memory via the Linux memory manager dma buf. The AMA memory management module manages the contiguous physical memory region, as expressed in the following formula: Treclaim (AMA size ratio) <Tallocate dma_buf(AMA size) In the above formula, AMA size is the total memory size of the contiguous physical memory region, ratio is a preset threshold, Tallocate dma_buf() is the total time taken to release memory through dma buf, and Treclaim() is the total time taken to release memory through the AMA memory management module.
[0058] In this embodiment, the large model file is pre-stored in a large model partition on the disk, and stored in the large model partition in a sequential order of consecutive sectors.
[0059] In a specific application embodiment, an AI Partition is provided, which uses contiguous sector storage to simplify the reading process and improve model loading speed. Furthermore, this invention adds a priority to the kernel I / O caller to reduce the queuing time for reading large model files. The solution is as follows: Create a separate partition (AI partition) on the disk and store the large model file in contiguous sectors within this partition; When reading files, large model files are constructed into bios in 128M units (that is, 250,000 sectors), which greatly reduces the number of bios and constructs the same number of requests. A new scheduling class IOPRIO_CLASS_AI has been added to the IO scheduler to increase the priority of AI application requests, giving them priority over ordinary applications in disk read and write processing. The format of the AI partition is as follows:
[0060] When an AI application begins loading a large model file, it first needs to call mmap to map the AMA memory region to the AI application's virtual memory address space before it can load the large model file from the file system into the AMA memory region. In this embodiment, as... Figure 8 As shown, when mapping the reserved contiguous physical memory region to the application's virtual memory address space, mmap is specifically used for mapping, and the steps are as follows: Iterate through each bit in the bitmap array corresponding to the contiguous physical memory region and determine whether it is 0. If it is not 0, further determine the memory page type. If it is an anonymous page, use swap partitioning. If it is a dirty page, write it back to the file system or discard it to realize memory reclamation or migration. After recycling or migration is complete, clear the bitmap array, set the large model file to start loading, map base_addr to the application's virtual memory address space, and return the virtual address.
[0061] It's understandable that AMA memory management uses reserved large blocks of memory and mmap address mapping to save time on allocating small blocks of memory, control memory overhead, and improve the loading efficiency of large model files. Furthermore, when large model files are not loaded, AMA can allocate a certain proportion of memory within the region to other modules, improving memory utilization. Specifically, the AI application calls mmap to map the AMA memory region to the application's virtual memory address space. After mapping, the AI application's virtual memory layout is as follows...Figure 9 As shown.
[0062] After memory usage is complete, such as Figure 10 As shown, alloc_free is used to release memory and clean up the corresponding bitmap.
[0063] Compared with the prior art, the present invention has the following beneficial effects: By using a fast allocation method for contiguous physical memory, the loading efficiency of large model files is improved, enhancing the user experience. Simultaneously, to avoid resource waste and improve memory utilization efficiency, this area can still be used by other kernel modules when no large model files are being loaded. However, a ratio variable is provided to ensure that the performance overhead of memory reclamation and migration is less than that of traditional memory allocation.
[0064] Sequential storage reduces the number of disk read requests and file system seek time. Furthermore, adding IOPRIO_CLASS_AI to the IO scheduler prioritizes AI application requests over those of regular applications, reducing queuing time.
[0065] The present invention further provides a system for fast loading of large model files under Linux, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute a method for fast loading of large model files under Linux.
[0066] The present invention further provides a computer-readable storage medium storing a computer program / instruction, the computer program / instruction being programmed or configured to execute a method for fast loading of large model files under a Linux system via a processor.
[0067] The system and medium of the present invention, corresponding to the methods described above, also have the advantages described above.
[0068] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. Computer-readable media include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.
[0069] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. A method for fast loading of large model files under a Linux system, characterized in that, The method comprises the steps of: When the Linux system starts, a continuous physical memory region with a preset size for loading a large model file is reserved according to the size of the large model file to be read; When an application program in the Linux system needs to load the large model file, the continuous physical memory region is mapped to the virtual memory address space of the application program, and the large model file is preferentially loaded into the continuous physical memory region, and the memory region occupied by the large model file is released after the large model file is used; When other modules in the Linux system request memory allocation, if it is detected that the continuous physical memory region is currently not loaded with the large model file and the sum of the requested memory size and the size of the memory already allocated in the continuous physical memory region is less than a preset threshold, the memory with the requested memory size is allocated to the corresponding module in the continuous physical memory region, otherwise, the memory is not allocated.
2. The method for fast loading of large model files under Linux system according to claim 1, characterized in that, The method further comprises the steps of: If it is detected that the large model file needs to be loaded and the continuous physical memory is insufficient for allocation, all the memory allocated to other modules of the Linux system in the continuous physical memory region is released for use by the large model file.
3. The method for fast loading of large model files under Linux system according to claim 1, characterized in that, Reserving a continuous physical memory region with a preset size for loading a large model file according to the size of the large model file to be read, specifically comprising: Configuring the size and starting address of the reserved memory region in the kernel startup parameter, or configuring the size, starting address and ending address of the reserved memory region in the device tree, to reserve a continuous physical memory region with a preset size.
4. The method for fast loading of large model files under a Linux system according to claim 1, characterized in that, Preferentially loading the large model file into the continuous physical memory region, specifically comprising: Adding a large model file scheduling class with a priority between a real-time task scheduling class and a normal application scheduling class in the IO scheduler of the Linux system; When the application program needs to load the large model file, a plurality of read-write requests are initiated according to a large model partition where the large model file is located with a preset length, and the scheduling classes corresponding to the plurality of read-write requests are set to the large model file scheduling class, so that the read-write priority of the large model file during loading is higher than the read-write priority of normal applications; The IO scheduler sends all the read-write requests to the disk driver according to the priority order, so that the disk driver writes the large model file into the continuous physical memory region according to the plurality of read-write requests.
5. The method for fast loading of large model files under Linux system according to claim 4, characterized in that, The preset length is 128M, corresponding to 250000 sectors.
6. The method for fast loading of large model files under Linux system according to claim 1, characterized in that, The preset threshold satisfies the following condition: When the large model file starts to be loaded, the total time consumed by the AMA memory management module for releasing the memory already allocated in the continuous physical memory region is less than the total time consumed by the Linux memory manager dma buf for memory release, wherein the AMA memory management module is used to manage the continuous physical memory region.
7. The method for fast loading of large model files under Linux system according to claim 6, characterized in that, The preset threshold satisfies the following formula: T reclaim (AMA size · ratio) < T allocate dma_buf (AMA size) In the above formula, AMA size is the total memory size of the continuous physical memory region, ratio is a preset threshold, T allocate dma_buf () is the total time consumption of memory release through the dma buf, T reclaim () is the total time consumption of memory release through the AMA memory management module.
8. The method for fast loading of large model files under Linux system according to any one of claims 1-7, characterized in that, The large model file is pre-stored in a large model partition of a disk, and is stored in a continuous sector order in the large model partition.
9. The method for fast loading of large model files under Linux system according to any one of claims 1-7, characterized in that, When the reserved continuous physical memory region is mapped to the virtual memory address space of the application program, the mapping is specifically performed by using mmap, and the steps are as follows: Each bit in the bitmap array corresponding to the continuous physical memory region is traversed and determined whether it is 0, if not 0, further determine the memory page type, if it is an anonymous page, take the swap partition mode, if it is a dirty page, write back to the file system or discard processing, to realize the recycling or migration of memory; After the recycling or migration is completed, the bitmap array is emptied, the large model file is set to start loading, the base_addr is mapped to the virtual memory address space of the application program, and the virtual address is returned.
10. A large model file fast loading system under a Linux system, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to perform the large model file fast loading method under the Linux system in any one of claims 1-9.