A terminal device-oriented large model dynamic loading inference method and system

Abstract

The application relates to the technical field of model reasoning, and discloses a large model dynamic loading reasoning method and system for terminal equipment, which comprises the following steps: in an offline planning stage, performing heterogeneous device feature analysis to evaluate the computing performance and I / O bandwidth of a target device and collecting model weight information; based on the feature analysis, an optimal splitting strategy is calculated through an adaptive scheduling algorithm, and the optimal splitting strategy is expressed as a tuple (k, w), wherein k represents the number of permanent resident decoder layers stored in the memory, and w represents the sliding window size of the dynamically loaded layers; in an online reasoning stage, based on the optimal splitting strategy, a hierarchical weight arrangement mechanism is adopted, the weights of the large language model are dynamically loaded and released through zero-copy I / O-computing parallel technology, and reasoning is performed.

Description

Technical Field

[0001] This invention relates to the field of model inference technology, and in particular to a method and system for dynamic loading and inference of large models for terminal devices. Background Technology

[0002] With the rapid development of edge intelligence, large language model (LLM) inference on devices is becoming a key enabling technology for mobile and edge AI systems due to its low latency response, data privacy protection, and personalized service capabilities. However, the continuous scaling of large language models and the stagnant memory capacity of mobile devices have created an increasingly acute contradiction, leading to a severe memory bottleneck for large language model inference on devices. Current mainstream memory optimization techniques mainly follow the approaches of server-side or traditional deep neural networks (DNNs), which are insufficient to effectively address the unique challenges of large language model inference on devices. First, the optimization objectives are misaligned. Existing work generally focuses on reducing the memory footprint of key-value (KV) caches, a strategy derived from server deployment experience in long-context scenarios. However, in typical mobile application scenarios (input length usually does not exceed 4K tokens), KV caches account for less than 15% of peak memory, while model weights themselves occupy more than 80% of the memory space. Therefore, optimizations targeting KV caches have little effect on mobile devices and cannot alleviate core memory pressure.

[0003] Secondly, the sparsity of weight access is not effectively utilized. Although the decoder layer accesses weights layer by layer in strict temporal order during inference, and only a very small portion of the token embedding matrix needs to be accessed in actual inference (for example, up to 78% of the embedding vectors in the Qwen model are never used), existing frameworks (such as llama.cpp) still adopt a "full preloading" strategy, loading all weights into memory at once, resulting in a large amount of memory waste.

[0004] Third, the operating system's memory management mechanism is neglected. Most systems rely on OS primitives such as `read` or `mmap` to load model weights, but fail to consider their underlying page cache behavior. This results in weight data residing simultaneously in both the kernel page cache and user space buffers, creating a "double buffering" effect, with actual memory usage reaching 1.5–2 times the theoretical value. More seriously, many benchmarks do not account for page cache overhead, leading to a significant overestimation of memory optimization effectiveness.

[0005] Fourth, heterogeneous deployment strategies lack adaptability. Traditional DNN layer offloading methods assume simple model structures and homogeneous hardware capabilities, while LLMs have deep stacked structures, and different mobile platforms (such as high-end mobile phones and embedded development boards) differ significantly in I / O bandwidth and computing power. Static layer partitioning strategies cannot be dynamically adjusted according to hardware characteristics, resulting in ineffective overlap between I / O and computation, and low resource utilization.

[0006] In summary, existing technologies fail to collaboratively optimize memory management across three dimensions: LLM access patterns, operating system behavior, and hardware heterogeneity. This severely restricts the stable and efficient deployment of large-scale models (7B–13B) on resource-constrained devices. Therefore, a new mechanism and architecture are urgently needed to systematically address these issues. Summary of the Invention

[0007] This invention provides a method and system for dynamic loading and reasoning of large models for terminal devices, in order to solve the problem of low positioning accuracy in existing large language model reasoning methods.

[0008] To achieve the above objectives, the present invention employs the following technical solution: In a first aspect, the present invention provides a method for dynamic loading and inference of large models for terminal devices, comprising: Offline planning phase: Perform heterogeneous device characteristic analysis to evaluate the computing performance and I / O bandwidth of the target device, and collect model weight information; Based on the feature analysis, the optimal segmentation strategy is calculated using an adaptive scheduling algorithm. The optimal segmentation strategy is represented as a tuple (k, w), where k represents the number of resident decoder layers permanently stored in memory, and w represents the sliding window size of the dynamically loaded layer. Online inference phase: Based on the optimal segmentation strategy, a hierarchical weight orchestration mechanism is adopted, and the weights of the large language model are dynamically loaded and released through zero-copy I / O-computation parallel technology for inference.

[0009] Optionally, the heterogeneous device feature analysis includes: Available memory capacity of the detection device; Measure storage bandwidth; The weights and computation time of each decoder layer in the statistical model; A database of hardware and model feature parameters is constructed to serve as input for the adaptive scheduling algorithm.

[0010] Optionally, the adaptive scheduling algorithm adopts a multi-armed slot machine model, treating each candidate strategy (k, w) as an arm, and using the upper confidence bound (UCB) algorithm to balance exploration and utilization during the iteration process, with the goal of minimizing inference latency, and selecting the optimal strategy.

[0011] Optionally, the reward score of the UCB algorithm is defined as a negative of the inference latency: r_{k,w} = -T(k, w), where T(k, w) is the estimated inference latency per token; The formula for calculating UCB scores is: ; in, For average reward, denoted by , where is the number of times the strategy was selected, and c is the exploration coefficient.

[0012] Optionally, the hierarchical weight orchestration mechanism includes: The weights of word embeddings are loaded on demand, and loading is only triggered by an operating system page error when the corresponding word is accessed for the first time. A sliding window execution strategy is adopted for the weights of the decoder layer. The first k layers are resident in memory, and the remaining layers are dynamically prefetched and evicted through a window of size w. Set the output projection weights to be permanently resident in memory.

[0013] Optionally, the zero-copy I / O-computation parallel technique includes: Construct a weight page table to map the logical address of the weights of each decoder layer to the physical location on disk, providing a continuous weight view; Zero-copy access is achieved using memory mapping, avoiding data copying; The producer-consumer synchronous model is adopted. The I / O thread asynchronously prefetches weights and sets an atomic ready flag, and the computation thread checks the flag and then performs the computation.

[0014] Optionally, the producer-consumer synchronization mode specifically includes: The I / O thread is responsible for prefetching the weights of subsequent dynamic layers and discarding layers that have already been used. The computation thread executes the decoding operation sequentially. Before each layer is executed, the atomic ready flag is checked. If the atomic ready flag is not ready, the thread blocks and waits. The memory usage and I / O-computation overlap can be dynamically adjusted by sliding the window size.

[0015] Optionally, the optimal partitioning strategy (k, w) must satisfy memory constraints: ; Among them, s i This represents the weight size of the i-th layer, and M is the available memory capacity of the device.

[0016] In a second aspect, embodiments of this application provide a large-model dynamic loading inference system for terminal devices, used to implement the method described in any one of the first aspects, comprising: The offline planning module is used to perform heterogeneous device feature analysis and adaptive scheduling algorithms to generate the optimal splitting strategy; An online inference module is used to implement hierarchical weight orchestration and zero-copy I / O-computation parallelism based on the aforementioned strategy; The memory management module is used to manage the loading, releasing, and page table mapping of weights.

[0017] A terminal device, characterized in that it integrates the system of claim 9, for efficiently running large language model inference in a resource-constrained environment.

[0018] Beneficial effects: The large-model dynamic loading and inference method for terminal devices provided by this invention effectively solves the memory bottleneck problems caused by redundant weight loading, unutilized access sparsity, and insufficient hardware heterogeneity adaptation in large language model inference on the device side through three core technologies: hierarchical weight management, zero-copy I / O-computational parallelism, and hardware adaptive scheduling. Experiments show that when running 7B–13B parameter models on mobile devices with 8GB of memory (such as Pixel 7 Pro and Raspberry Pi 5), H2O can reduce peak memory usage by up to 60%, completely eliminating memory overflow failures. At the same time, under strict memory constraints, the inference latency is reduced by 34%–94% compared to mainstream baselines (such as LLAMA-Mmap and On-Demand), and CPU utilization is increased to over 50% (compared to baselines of 25%–30%). This solution supports dynamic reconfiguration strategies within 85ms to adapt to memory fluctuations without interrupting inference, significantly improving the deployment feasibility and user experience of large models on resource-constrained devices, and providing cost-effective and reliable technical support for privacy-sensitive, low-latency edge AI applications (such as mobile assistants and edge agents). Attached Figure Description

[0019] Figure 1 A flowchart illustrating the workflow of the heterogeneous sensing hierarchical orchestration framework designed for this invention; Figure 2 This is a comparison diagram of parallel I / O computation proposed in this invention and simple synchronous I / O computation; Figure 3 This is a comparison chart of peak memory usage and inference latency when deploying and running the llama2-7b-chat-q8 quantization model using a Raspberry Pi 5 in a preferred embodiment of the present invention.

[0020] Figure 4 This is a comparison chart of CPU utilization and inference latency when deploying and running the llama2-7b-chat-q8 quantization model using a Raspberry Pi 5 in a preferred embodiment of the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms "an" or "a" and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms "connected" or "linked" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0023] Please see Figure 1 This application provides a method for dynamic loading and inference of large models for terminal devices. This embodiment uses the deployment and running of the llama2-7b-chat-q8 quantized model (total model size approximately 6.8 GB, containing 32 decoder layers) on a Raspberry Pi 5 development board (equipped with 8 GB RAM and a quad-core ARM Cortex-A76 CPU) as an example to detail the specific implementation steps of the invention. The entire implementation process is divided into two stages: offline planning and online inference. Its overall workflow is as follows: Figure 1 As shown.

[0024] Step 1: Heterogeneous Device Analysis (Offline Profiling) When the model is first deployed or the system detects a significant change in the hardware environment (such as changing storage media or memory being heavily used by other processes), the system automatically triggers an offline hardware analysis process to build an accurate performance profile for the current device. Memory capacity detection: The system reads the Linux kernel interface / proc / meminfo to obtain the total amount of physical memory currently available on the device, and combines this with the system operating overhead to reserve a safety margin, ultimately determining that the available memory budget for the inference task is M = 7.2GB.

[0025] Storage bandwidth benchmark: The system performs a lightweight I / O micro-benchmark test—by sequentially reading a temporary file larger than 1 GB, the actual sequential read bandwidth of the MicroSD card used by the device is measured and denoted as B. This parameter directly determines the speed at which weights are loaded from storage to memory.

[0026] Model layer analysis: The system loads the model's metadata file, and can accurately calculate the weight size {si} of each decoder layer without loading the complete weights. It also estimates the computation time {tcomp} of each layer on the current CPU by performing pre-inference on a small number of sample tokens.

[0027] Build an analysis database: All the hardware and model feature parameters collected above (including M, B, {si}, {tcomp} and the total number of model layers L=32) are structured and stored in a local configuration file as input for the subsequent adaptive scheduling algorithm.

[0028] Step 2: Adaptive scheduling algorithm generates optimal execution plan Based on the hardware profile collected in step one, the system runs an adaptive scheduling algorithm to find the hierarchical execution strategy (k, w) that satisfies memory constraints and minimizes inference latency. In this strategy, k represents the number of decoder layers that need to reside in memory, and w represents the size of the sliding window used for dynamic loading. The specific implementation process is as follows: 1. Constructing the Candidate Pool: The algorithm first enumerates all possible (k, w) combinations within the theoretically feasible region (e.g., for a 32-layer model, k∈[0,32], w∈[1,16]), forming an initial candidate space (which can reach hundreds of entries). To improve search efficiency and eliminate invalid configurations, the system applies a triple filtering mechanism in sequence: Memory constraint filtering: based on the formula:

[0029] Eliminate all configurations with peak memory exceeding the 7.2 GB budget; Analysis-guided pruning: Based on the statistical distribution of the weights and computation time of each layer, dynamic strategies that have excessively large w values ​​(leading to a surge in I / O pressure) or excessively small w values ​​(causing frequent computational blocking) are dynamically eliminated. Feasibility simulation: Lightweight runtime memory state simulation is performed on the remaining candidates to verify whether their performance may be degraded in actual inference due to frequent page faults or scheduling conflicts.

[0030] Through this triple screening, the candidate pool is efficiently compressed into a subset of approximately 20–30 high-potential, high-feasibility strategies, laying the foundation for subsequent optimization.

[0031] 2. Multi-Armed Slot Machine (MAB) Optimization: The system treats each candidate strategy as an "arm" in a multi-armed slot machine, employing the UCB1 (Upper Confidence Bound) algorithm for intelligent exploration and utilization. In approximately 50 iterations, each time based on the UCB score:

[0032] Choose the current optimal strategy; The analysis model is invoked to quickly estimate the per-token inference latency T(k,w) under the selected (k,w) strategy. This model comprehensively considers the computation time of the resident layer, the I / O loading time of the dynamic layer, the release overhead of the inner layer of the sliding window, and the I / O computation synchronization latency. Negative delay: The UCB mechanism provides immediate rewards to update the statistics of the strategy, thereby achieving a dynamic balance between "utilizing known efficient strategies" and "exploring potential better solutions".

[0033] 3. Output Optimal Strategy: After iteration, the algorithm selects the strategy with the highest historical average reward as the final execution plan. In this embodiment, considering the Raspberry Pi 5's low I / O bandwidth and 7.2GB memory limit, the optimal strategy output by the algorithm is (k=8, w=4). The specific meaning of this strategy is as follows: The weights of the first 8 decoder layers are permanently loaded into memory during the inference initialization phase to avoid computational stagnation caused by I / O waits in the early stages of autoregressive generation. The remaining 24 layers are dynamically managed using a sliding window of size 4. The I / O thread asynchronously prefetches the weights of the 4 layers to be computed in the background, and releases the oldest layer that has been computed in a timely manner when the window slides, ensuring that memory usage is always below the 7.2GB limit. This strategy has been verified by the analysis model to achieve the lowest end-to-end inference latency while meeting memory constraints. Step 3: Hierarchical weight orchestration and zero-copy parallelism in the online inference phase The system initiates online inference based on the generated plan (k=8, w=4): 1. Initialization phase: Resident layer loading: The weights of the first 8 decoders and the output projection weights (approximately 1.5GB) are mapped to virtual memory via mmap, but the data is not loaded immediately; only the virtual addresses are recorded.

[0034] Construct a weighted page table: Create a "weighted page table" to accurately record the offset position and size of each weight in the disk file, providing a logical-to-physical mapping for subsequent zero-copy access.

[0035] Start the I / O and computation threads: Create a dedicated I / O thread to handle weight prefetching and a computation thread to handle inference.

[0036] 2. Handling user input (Prompt): The user inputs the prompt "Hello, how are you?".

[0037] Lexical embeddings are loaded on demand: The system segments the prompt words into [Hello, how, are, you, ?]. When calculating the embedding vector for the first lexical "Hello", an operating system page fault is triggered, and the system only loads the embedding vector corresponding to "Hello" (approximately a few KB) from the disk, instead of the entire 788MB embedding matrix. Subsequent lexical embeddings are handled similarly, thus saving memory.

[0038] 3. Autoregressive generation stage (taking the generation of the first new lexical as an example): I / O thread operation: According to the sliding window strategy (w=4), the I / O thread asynchronously prefetches the weight data of layers 9, 10, 11, and 12 into memory (on-demand loading triggered by mmap), and sets an "atomic ready flag" for each layer.

[0039] Thread computation work: Perform resident first 8 layer decoder calculations (no I / O wait).

[0040] When the 9th level needs to be executed, its "ready flag" is checked. If the I / O thread has finished loading (flag is True), it executes immediately; otherwise, it is briefly blocked and waits.

[0041] After execution, the I / O thread is notified that the layer has been "consumed" and can be replaced by subsequent layers.

[0042] Sliding window advancement: The operation process is as follows Figure 2 As shown, the green thread represents the calculation thread. When the weight of the first layer in the sliding window is calculated, the IO thread will immediately release the C1 layer (the 9th layer) that has just been calculated, and calculate the next layer to be prefetched, L5 (the 13th layer). While the calculation thread is calculating the 10th layer, the weight of the 13th layer is loaded at the same time, so that there are always 4 layers of dynamic weights in the window.

[0043] 4. Output projection: For each word generated, the computation thread accesses the complete output projection weights (which are already resident in memory), performs Softmax calculation, and outputs the probability distribution of the next word.

[0044] This application also provides a large model dynamic loading inference system for terminal devices, which implements any one of the methods described in the large model dynamic loading inference method for terminal devices, including: The offline planning module is used to perform heterogeneous device feature analysis and adaptive scheduling algorithms to generate the optimal splitting strategy; An online inference module is used to implement hierarchical weight orchestration and zero-copy I / O-computation parallelism based on the aforementioned strategy; The memory management module is used to manage the loading, releasing, and page table mapping of weights.

[0045] A terminal device, characterized in that it integrates the system of claim 9, for efficiently running large language model inference in a resource-constrained environment.

[0046] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for dynamic loading and inference of large models for terminal devices, characterized in that, include: Offline planning phase: Perform heterogeneous device characteristic analysis to evaluate the computing performance and I / O bandwidth of the target device, and collect model weight information; Based on the feature analysis, the optimal segmentation strategy is calculated using an adaptive scheduling algorithm. The optimal segmentation strategy is represented as a tuple (k, w), where k represents the number of resident decoder layers permanently stored in memory, and w represents the sliding window size of the dynamically loaded layer. Online inference phase: Based on the optimal segmentation strategy, a hierarchical weight orchestration mechanism is adopted, and the weights of the large language model are dynamically loaded and released through zero-copy I / O-computation parallel technology for inference. The hierarchical weighting mechanism includes: The weights of word embeddings are loaded on demand, and loading is only triggered by an operating system page error when the corresponding word is accessed for the first time. A sliding window execution strategy is adopted for the weights of the decoder layer. The first k layers are resident in memory, and the remaining layers are dynamically prefetched and evicted through a window of size w. Set the output projection weights to be permanently resident in memory; The zero-copy I / O-computation parallel technology includes: Construct a weight page table to map the logical address of the weights of each decoder layer to the physical location on disk, providing a continuous view of weights; Zero-copy access is achieved using memory mapping, avoiding data copying; The producer-consumer synchronous model is adopted. The I / O thread asynchronously prefetches weights and sets an atomic ready flag, and the computation thread checks the flag and then performs the computation.

2. The method for dynamic loading and inference of large models for terminal devices according to claim 1, characterized in that, The heterogeneous device feature analysis includes: Available memory capacity of the detection device; Measure storage bandwidth; The weights and computation time of each decoder layer in the statistical model; A database of hardware and model feature parameters is constructed to serve as input for the adaptive scheduling algorithm.

3. The method for dynamic loading and inference of large models for terminal devices according to claim 1, characterized in that, The adaptive scheduling algorithm adopts a multi-armed slot machine model, treating each candidate strategy (k,w) as an arm. It uses the upper confidence bound UCB algorithm to balance exploration and utilization during the iteration process, aiming to minimize inference latency and select the optimal strategy.

4. The method for dynamic loading and inference of large models for terminal devices according to claim 3, characterized in that, The reward score of the UCB algorithm is defined as a negative of the inference latency: r_{k,w} = -T(k,w), where T(k,w) is the estimated inference latency per token; The formula for calculating UCB scores is: ； in, For average reward, The number of times the strategy was selected. To explore coefficients.

5. The method for dynamic loading and inference of large models for terminal devices according to claim 1, characterized in that, The producer-consumer synchronization model specifically includes: The I / O thread is responsible for prefetching the weights of subsequent dynamic layers and discarding layers that have already been used. The computation thread executes the decoding operation sequentially. Before each layer is executed, the atomic ready flag is checked. If the atomic ready flag is not ready, the thread blocks and waits. The memory usage and I / O-computation overlap can be dynamically adjusted by changing the size of the sliding window.

6. The method for dynamic loading and inference of large models for terminal devices according to claim 1, characterized in that, The optimal segmentation strategy (k, w) must satisfy the memory constraint: ； in, Indicates the first Layer weight size, This represents the available memory capacity of the device.

7. A large-model dynamic loading inference system for terminal devices, used to implement the method of any one of claims 1-6, characterized in that, include: The offline planning module is used to perform heterogeneous device feature analysis and adaptive scheduling algorithms to generate the optimal splitting strategy; An online inference module is used to implement hierarchical weight orchestration and zero-copy I / O-computation parallelism based on the aforementioned strategy; The memory management module is used to manage the loading, releasing, and page table mapping of weights.

Images